Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Next »

 Watch video

About the autoparser

When data enters Devo through a non-standardized method, it cannot be automatically parsed at that moment of reception due to the lack of Devo tags to interpret its structure. However, it can still be parsed automatically thanks to the Autoparser. This functionality analyzes the data patterns in search of possible ways to parse your data, allowing you to choose the one you consider most adequate to resemble the originally intended structure.

How does it work?

The Autoparser uses internal logic to analyze the content of up to 200 sample events in an unparsed table in order to identify:

  • Non-alphanumeric ASCII characters that might be field delimiters. 

  • Units of data that follow a common and fixed syntax; i.e. IPv4 and IPv6 addresses, strings enclosed in quotation marks, floating-point decimal values, and more.

Based on this analysis, it recommends a selection of characters that are likely (and less likely) to be delimiters. You can then use the Autoparser controls to select the delimiters you need or analyze a different sample.

Delimiter analysis

The Autoparser will only offer as delimiters those characters that appear in all the 200 logs analyzed. For example, the comma will not be offered as a delimiter in the following set because the third element does not have any:
a, b, c, d // x, y, z // sampleLog

What data do I need?

Tables

The table must be one of the following:

  • my.app → tables created by sending data from a new, proprietary data source.

  • my.upload → tables created by manually uploading a file containing data.

Unless…

  • They have less than 4 tag levels, which makes them ineligible for the use of the autoparser.

  • They have been created by injecting data from another table, which are already properly parsed.

Events

Events must comply with the following conditions:

  • Events must contain the same number of fields in the same order.

  • Events must use delimiters in the same pattern in every event.

  • Events must use valid formats.

    • Timestamp data will be parsed as a string when presenting an invalid format.

      • ddd MMM DD HH:mm:ss YYYY → Thu Mar 29 00:21:05 2012

      • DD-MMM-YYYY HH:mm:ss.SSS → 27-Aug-2012 09:44:09.378

      • DD/MM/YYYY HH:mm:ss → 23/07/2019 07:55:00

      • DD/MM/YYYY H:mm:ss → 14/09/2012 9:42:05

      • YYYY-MM-DD HH:mm:ss.SSS → 2000-12-17 01:01:01.123

      • YYYY/MM/DD HH:mm:ss → 2012/08/25 06:48:18

      • YYYY/MM/DD → 2012/08/25

      • yyyy-MM-dd:hh:mm:ss+gmt → 2012-08-16:10:29:17+0200

      • yyyy-MM-dd hh:mm:ss.micros → 2000-12-17 01:01:01.123456

      • Epoch.millis (recognized as a float, you need to change it) → 1234567890.123

      • Millis (recognized as an integer, you need to change it) → 1584100816544

Disparate data

If the events do not conform to these requirements, it may be imposible to extract a pattern and an error will pop up to further specify the reason. You can then parse it manually using operations such as Split (split) and later create a custom table.

For my.app data, you can also contact customer support to request a custom parser.

What permissions do I need?

To use this feature, you need two different permissions:

  • View level of the Finders permission. This permission is required to access a search, which is necessary to use the autoparser.

  • The Log autoparser permission. This permission is required to enable the option in the toolbar menu of the search window.

Using the Autoparser

The process of parsing will not create a new table but transform the original table instead so it can be fully used in Devo.

Autoparsing an already parsed table

If you are not happy with the result for whatever reason, you can use the Autoparser again. However, it is not possible to apply changes selectively; the table will be reset to its original unparsed state for you to start over.

You just need to open the Autoparser as instructed before. Don't worry if you have second thoughts after clicking because you will receive a warning message. You can either cancel and keep the table as it is or restore it to parse it again with different settings.

If you choose to restore it, you will be forced out while the table goes back to its original unparsed state. To parse it anew, you need to access it again through the Data Search and open the Autoparser once more.

Autoparsing special objects

It is possible to autoparse special objects such as JSON. Although the concept is the same, the procedure varies. Check the article Autoparse a JSON object to know more.

  • No labels