Autoparser

[ 1 About the autoparser ] [ 2 How does it work? ] [ 3 What data do I need? ] [ 4 What permissions do I need? ] [ 5 Using the autoparser ]

About the autoparser

When data enters Devo through a non-standardized method, it cannot be automatically parsed at that moment of reception due to the lack of Devo tags to interpret its structure. However, it can still be parsed automatically thanks to the Autoparser. This functionality analyzes the data patterns in search of possible ways to parse your data, allowing you to choose the one you consider most adequate to resemble the originally intended structure.

How does it work?

The Autoparser uses internal logic to analyze the content of up to 200 sample events in an unparsed table in order to identify:

Non-alphanumeric ASCII characters that might be field delimiters.
Units of data that follow a common and fixed syntax; i.e. IPv4 and IPv6 addresses, strings enclosed in quotation marks, floating-point decimal values, and more.

Based on this analysis, it recommends a selection of characters that are likely (and less likely) to be delimiters. You can then use the Autoparser controls to select the delimiters you need or analyze a different sample.

Delimiter analysis

The Autoparser will only offer as delimiters those characters that appear in all the 200 logs analyzed. For example, the comma will not be offered as a delimiter in the following set because the third element does not have any:
a, b, c, d // x, y, z // sampleLog

Invalid delimiters

The Autoparser never selects wrappers such as (), [], {}, "", '' as delimiters, nor does it choose delimiters inside them.

What data do I need?

Tables

The table must be one of the following:

my.app → tables created by sending data from a new, proprietary data source.
my.upload → tables created by manually uploading a file containing data.

Unless…

They have less than 4 tag levels, which makes them ineligible for the use of the autoparser.
They have been created by injecting data from another table, which are already properly parsed.

Events

Events must comply with the following conditions:

Events must contain the same number of fields in the same order.
Events must use delimiters in the same pattern in every event.
Events must use valid formats.
Timestamp data will be parsed as a string when presenting an invalid format. Check valid formats below:
- ddd MMM DD HH:mm:ss YYYY → Thu Mar 29 00:21:05 2012
- DD-MMM-YYYY HH:mm:ss.SSS → 27-Aug-2012 09:44:09.378
- DD/MM/YYYY HH:mm:ss → 23/07/2019 07:55:00
- DD/MM/YYYY H:mm:ss → 14/09/2012 9:42:05
- YYYY-MM-DD HH:mm:ss.SSS → 2000-12-17 01:01:01.123
- YYYY/MM/DD HH:mm:ss → 2012/08/25 06:48:18
- YYYY/MM/DD → 2012/08/25
- yyyy-MM-dd:hh:mm:ss+gmt → 2012-08-16:10:29:17+0200
- yyyy-MM-dd hh:mm:ss.micros → 2000-12-17 01:01:01.123456
- Epoch.millis (recognized as a float, you need to change it) → 1234567890.123
- Millis (recognized as an integer, you need to change it) → 1584100816544

Disparate data

If the events do not conform to these requirements, it may be impossible to extract a pattern and an error will pop up informing you that your data has been parsed with errors. This means that some fields could not be parsed properly and you will see null values wherever the error occurred.

To avoid this, you can parse this data manually using operations such as Split (split) and later create a custom table. For my.app data, you can also contact customer support to request a custom parser.

What permissions do I need?

To use this feature, you need two different permissions:

View level of the Finders permission. This permission is required to access a search, which is necessary to use the autoparser.
The Log autoparser permission. This permission is required to enable the option in the toolbar menu of the search window.