Special Devo tags and data tables

This article explains the difference between tags and tables and describes some of the special tags used for sending data to Devo and the tables you might see in your finder.

About tags and tables

In Devo, tags are used primarily for ingesting and storing data sent from a source system to Devo, while tables are used to retrieve, parse, and display data saved in Devo. In effect, a table offers a view of the data that you've saved. For example, it's possible to create two tables that retrieve data from the same file but display that data differently. One could display fewer columns or display the columns in a different order. Similarly, a table can be created to draw data from multiple files that contain similar data in a similar structure. This is called a parent table and an example would be the firewall.fortinet.traffic table. You can read more about them later in this article. 

Most of the time, a tag and its resulting table have the same name, as is the case with box.win or cms.wordpress.stdout. However, there are a number of tags that contain hyphens, a character not allowed in table names. In these cases, the table name differs from the tag and uses camel case instead. For example, the web.apache.access-lt tag corresponds to the web.apache.accessLt table in Devo. If you are ever in doubt, you can see the table's name in the operations' path of the search window like this:

Open

my.app

Tags and tables beginning with my.app can describe two types of data.

Events for which there is no Devo tag

When you want to send events to Devo from a source for which there is no Devo tag, you can create your own tag using my.app as the prefix. This can happen with proprietary data sources or publicly available sources for which Devo has not yet created a tag (and therefore, there's no associated parser). 

The full tag should have at least four levels and may have up to six. The first two must be my.app. The third and fourth levels are free and should describe the application type and event type respectively. The fifth and sixth levels are optional and should be used to identify the actual source of the events. For example, if there are several servers running the application and reporting events to Devo, these levels can help identify the event's specific event source. 

For example:

• my.app.salesapp.activities.usa.server02

• my.app.salesapp.activities.usa.server01

When Devo receives an event with a tag that begins with my.app, it saves the event to a file and location determined by the tag levels, adds the first four levels of the tag to the finder (this will be the table), and saves the fifth and sixth levels (if any) to the event in the data table. However, since Devo is not equipped with a parser for this event type, when you open the data table, each event row will have only a few fields:

• eventdate - This is the date/time the event was received by Devo.

• cluster - This is the fifth level of the tag (if it was used).

• instance - This is the sixth level of the tag (if it was used).

• message - This contains the unparsed content as it was received.

You can manually parse the content of the message field using the Autoparser tool in the search window, or using the column operations available (for example, creating new columns using the Split (split) operation). Then, when you've parsed all the fields into columns, create a custom table. From this point, you can use the custom table to consult the data parsed into columns.

Tables created by injecting existing data

When you inject data into a new table, the new table will always be assigned the prefix my.app. 

my.upload

Tags and tables beginning with my.upload  are created when uploading static files to Devo, either from your local machine or from a Dropbox account.

Uploading static files can be useful to log historical data as opposed to real-time data. These files will be stored in tables starting with the my.upload tag levels. The first two levels of the tag are predefined as my.upload. The third and fourth levels are up to you.

For more information about sending event files to a my.upload table, see Uploading log files.

my.lookuplist

Lookup tables created by uploading a .csv file or using query data are saved using the tag my.lookuplist.<lookup_name>.

There is one my.lookuplist.<lookup_name> table for each existing lookup. These are synthetic tables created to enable querying the content of each lookup, which is repeated every single day and spread over the whole day. For that reason, it is advisable to query these tables using an exact one-day time range – today()-1d <= eventdate < today().

Lookup tables enrich the information in raw data tables, added to the virtual data table at query time as new columns. The original data tables are never modified. For more information, see Data Enrichment.

test.keep.free

This tag is only used to test event sending after you have set up a new event source, Devo relay, or to test a relay rule. Events that are received with this tag are saved in the test.keep.free table which you can consult to confirm that your events have been sent correctly and with the correct data.

unknown.unknown

This is a table that Devo automatically generates when it receives any inbound event that has no tag or has a tag that it doesn't recognize. Instead of dropping the event, it is saved to this table. When this table appears in your finder, open it to investigate the nature of the problem, and address it. 

bad.illegal

When Devo receives an event that is not properly structured to an existing tag, it is sent to this table.

cef0.* tables

While we recommend sending data to Devo in syslog format whenever possible, we support the ingestion of events received in common event format (CEF) via syslog for some technologies. A prime example of the use of this particular format is when ArcSight is used as a log management solution and events are going to be forwarded from ArcSight directly to Devo.

When events are sent in CEF syslog format, they don't require a Devo tag to enable ingestion. However, it is necessary for the platform to be equipped with a parser in order for you to access the data tables and see the events parsed correctly at query time. 

Learn more about CEF syslog and check out the list of technologies that Devo currently supports when sent in this special format.

Custom tables

A table in the finder that begins with my.synthesis, my.parser or my.tech identifies a custom table; one that a user has created from another table's contents or by forming a union of two tables. When creating a custom table, the user can choose which prefix they prefer to use.

For more information about custom tables, see Custom and union tables.

Union tables

Union tables are special tables that collect information from different source tables and they may serve a variety of purposes. Users can create union tables to help them with their work and they can also check a set of union tables generated by Devo to provide certain information that may be of general interest. To know more about these tables, check the Union tables article. 

Parent tables

These are like union tables but collect multiple event types from the same technology in a single table. In other words, parent tables are like unions of their child tables and they are created automatically.

Let's look at an example. F5 Big-IP generates several different log types and Devo has created the following tags to save each in a separate table. Devo has created tags to ingest the audit, local traffic, packet filter, and system logs:

• network.f5.bigip.audit

• network.f5.bigip.ltm

• network.f5.bigip.pktfilter

• network.f5.bigip.system

If your deployment is collecting these events, you will see these tables in your finder (they have the same names as the tags used for ingestion). Open any of these tables to see the collection of events for the corresponding event type.

However, you will also find a parent table called network.f5.bigip. If you open this table, you will see all the events collected using the network.f5.bigip.* tags in a single view.  

Parent tables exist for several technologies but only when the different event formats and event collection policy allow for it. Here are some other examples of parent tables:

• firewall.fortinet

• box.win_nxlog

• network.cisco

• network.meraki

• av.sophos