Autoparser

Overview

When data enters Devo through a non-standardized method, it cannot be automatically parsed at that moment of reception due to the lack of Devo tags to interpret its structure. However, it can still be parsed automatically thanks to the Autoparser. This functionality analyzes the data patterns in search of possible ways to parse your data, allowing you to choose the one you consider most adequate to resemble the originally intended structure.

The following video is an introduction to the autoparser concept:

How does it work?

The autoparser uses an internal logic to analyze the content of up to 200 sample events in an unparsed table in order to identify:

• Non-alphanumeric ASCII characters that might be field delimiters. 
• Units of data that follow a common and fixed syntax; i.e. IPv4 and IPv6 addresses, strings enclosed in quotation marks, floating-point decimal values, and more.

Based on this analysis, it recommends a selection of characters that are likely (and less likely) to be delimiters. Using the autoparser controls, you can select the delimiters that you need and deselect the rest. By testing the same delimiter pattern on different sample events, you can confirm that it will parse your data table as needed.

What type of tables can be autoparsed?

The Autoparser will be available for the following types of tables:

• my.app → tables created by sending data from a new, proprietary data source. 
• my.upload → tables created by manually uploading a file containing data.

What data do I need for a successful autoparse?

In certain cases, the autoparser can be used to parse these data tables quickly and easily. However, for the autoparser to work optimally, the log events must:

• Contain the same number of fields in the same order.
• Use delimiters in the same pattern in every event.

Valid timestamp formats

When the table to be parsed contains timestamp data, it must present a valid format, otherwise it will be parsed as a string. Check the valid formats in the table below:

Disparate data

If the log events you need to parse do not conform to these requirements, the autoparser may not be your best way forward. These data results too disparate for the Autoparser to extract a pattern so it will not open and an error message will pop up to further specify the reason.

In those cases, you can manually parse the content of the message field using the column operations available (for example, creating new columns using the Split (split) operation). Then, you can create a custom table and use it to consult the data parsed into columns.

You can also contact customer support to request a custom parser for your my.app data.

Using the Autoparser

• Go to Data Search, and select the new table using the finder.

• Click the gear icon in the toolbar and select Source table → Autoparse.

  The option will not appear for you to select if the requirements explained in the type of tables section are not met.

  
• The autoparser window opens so you can select the desired settings to transform the raw data contained in the message column into a fully classified table.

• If you want a different set of samples to better analyze the adequacy of the delimiters you chose, click the Reload samples button  next to the No. of Samples dropdown. This will lead to two possible scenarios depending on the temporal aspects of your query:

  • Fixed period query: if you have the query running for a fixed period so new events are not being received, you will get a warning message telling you that "no new events were found" so changing the samples is not possible

  • Real-time query: if you have the query running in real-time so new events are being received, this will load the last 200. If the structure changes, new delimiters will be considered. If the structure is too different, you will get the "Disparate Data" error message explained before.

• Select or deselect the symbols identified as possible field Delimiters. You can either select them by clicking in the Delimiters area or one by one in the Sample area. The symbols are displayed in colors to show if they are going to be used when parsing:

  Color	Meaning
  	All the symbols in the sample are going to be used.
  	None of the symbols in the sample are going to be used.
  	Some of the symbols in the sample are going to be used. A number is displayed below to specify how many of them.

• Once you have selected the desired pattern of delimiters, you can assign names and select the required data type for each column using the dropdown menu whenever possible.
• Select the Exclude checkbox for any columns you do not want to include in the parsed table. When you do so, the corresponding name field will be automatically disabled.
• Click Confirm Settings.

Autoparsing an already parsed table

If you are not happy with the result for whatever reason, you can use the Autoparser again. However, it is not possible to apply changes selectively; the table will be reset to its original unparsed state for you to start over.

You just need to open the Autoparser as instructed before. Don't worry if you have second thoughts after clicking because you will receive a warning message. You can either cancel and keep the table as it is or restore it to parse it again with different settings.

If you choose to restore it, you will be forced out while the table goes back to its original unparsed state. To parse it anew, you need to access it again through the Data Search and open the Autoparser once more.

Autoparsing special objects

It is possible to autoparse special objects such as JSON. Although the concept is the same, the procedure varies. Check the article Autoparse a JSON object to know more.

.