Autoparser
- Juan Tomás Alonso Nieto (Deactivated)
Overview
When data enters Devo through a non-standardized method, it cannot be automatically parsed at that moment of reception due to the lack of Devo tags to interpret its structure. However, it can still be parsed automatically thanks to the Autoparser. This functionality analyzes the data patterns in search of possible ways to parse your data, allowing you to choose the one you consider most adequate to resemble the originally intended structure.
How does it work?
The autoparser uses internal logic to analyze the content of up to 200 sample events in an unparsed table in order to identify:
Non-alphanumeric ASCII characters that might be field delimiters.
Units of data that follow a common and fixed syntax; i.e. IPv4 and IPv6 addresses, strings enclosed in quotation marks, floating-point decimal values, and more.
Based on this analysis, it recommends a selection of characters that are likely (and less likely) to be delimiters. Using the autoparser controls, you can select the delimiters that you need and deselect the rest. By testing the same delimiter pattern on different sample events, you can confirm that it will parse your data table as needed.
Delimiter analysis
The Autoparser will only offer as delimiters those characters that appear in all the 200 logs analyzed. For example, in the series of logs below, the comma will not be offered as a delimiter since the third log does not include any.
a, b, c, d
x, y, z
sampleLog
What type of tables can be autoparsed?
The Autoparser will be available for the following types of tables:
my.app→ tables created by sending data from a new, proprietary data source.my.upload→ tables created by manually uploading a file (either locally or from a drive) containing data.
Not available for tables...
Be aware that the autoparser will not be available for the tables mentioned above unless they have at least four tag levels. It will be unavailable as well for these tables if they were created by injecting data from another table, since these are already properly parsed.
What data do I need for a successful autoparse?
In certain cases, the autoparser can be used to parse these data tables quickly and easily. However, for the autoparser to work optimally, the log events must:
Contain the same number of fields in the same order.
Use delimiters in the same pattern in every event.
Valid timestamp formats
When the table to be parsed contains timestamp data, it must present a valid format, otherwise, it will be parsed as a string. Check the valid formats in the table below:
Format | Example | Format | Example |
|---|---|---|---|
ddd MMM DD HH:mm:ss YYYY | Thu Mar 29 00:21:05 2012 | YYYY/MM/DD | 2012/08/25 |
DD-MMM-YYYY HH:mm:ss.SSS | 27-Aug-2012 09:44:09.378 | yyyy-MM-dd:hh:mm:ss+gmt | 2012-08-16:10:29:17+0200 |
DD/MM/YYYY HH:mm:ss | 23/07/2019 07:55:00 | yyyy-MM-dd hh:mm:ss.micros | 2000-12-17 01:01:01.123456 |
DD/MM/YYYY H:mm:ss | 14/09/2012 9:42:05 | Epoch.millis By default recognized as a float so it needs to be manually changed. | 1234567890.123 |
YYYY-MM-DD HH:mm:ss.SSS | 2000-12-17 01:01:01.123 | Millis By default recognized as an integer so it needs to be manually changed. | 1584100816544 |
YYYY/MM/DD HH:mm:ss | 2012/08/25 06:48:18 |
Disparate data
If the log events you need to parse do not conform to these requirements, the autoparser may not be your best way forward. These data may result too disparate for the Autoparser to extract a pattern so it will not open and an error message will pop up to further specify the reason.
In those cases, you can manually parse the content of the message field using the column operations available (for example, creating new columns using the Split (split) operation). Then, you can create a custom table and use it to consult the data parsed into columns.
For my.app data, you can also contact customer support to request a custom parser.
Using the Autoparser
The process of parsing will not create a new table but transform the original table instead so it can be fully used in Devo.
Autoparsing an already parsed table
If you are not happy with the result for whatever reason, you can use the Autoparser again. However, it is not possible to apply changes selectively; the table will be reset to its original unparsed state for you to start over.
You just need to open the Autoparser as instructed before. Don't worry if you have second thoughts after clicking because you will receive a warning message. You can either cancel and keep the table as it is or restore it to parse it again with different settings.
If you choose to restore it, you will be forced out while the table goes back to its original unparsed state. To parse it anew, you need to access it again through the Data Search and open the Autoparser once more.
Autoparsing special objects
It is possible to autoparse special objects such as JSON. Although the concept is the same, the procedure varies. Check the article Autoparse a JSON object to know more.