Tags are at the heart of what makes Devo such a powerful and efficient data operations platform.
Devo uses tags to identify each event's data source and store them in the repository in the same file as all the other events that share the same tag. When events arrive to the platform, each one is assigned a timestamp, or eventdate, used to order events chronologically as they are received. Events are then saved in their original, unparsed format, to a directory and file that is based upon the tag they arrive with.
Analysts working with Devo later use the tag as a way of identifying the log file they want to query using the Finder. When a tag is selected in the Finder, the events in the file are parsed for display in the query window, using the parser associated with that tag.
Devo has pre-defined tags for hundreds of common data sources. This means that Devo is already capable of correctly parsing the data source's events. For more details, read about the supported technologies.
This article describes the following topics related to Devo tags:
How events are tagged
Tags can be applied to events in a few different ways. The important thing is that all events that are delivered to the Devo Cloud are associated with the correct tag. Tags can be associated to events:
At the data source. For example, rsyslog offers the ability to associate tags to the data it sends.
By the Devo Relay. Tags are applied by rules that process incoming events depending on the relay entry port.
The structure of a tag
In Devo, tags are composed of multiple elements (or levels), a minimum of two and a maximum of six, separated by dots following this general format:
The two first elements, technology and brand, are mandatory and help to identify and categorize the data source. For example, web.apache or web.nginx are different tag elements used to describe the data sources as web servers, and the brands indicate the vendor.
The type and subtype elements are optional but you will find that in most cases, either one or both are needed to describe the specific data source (log file). Continuing our example, both Apache and Nginx web servers generate logs of several types. As a result, the tags for the Nginx access and error logs are web.nginx.accesslt and web.nginx.error. For an Apache server, these are web.apache.accesslt and web.apache.error. Now, a company may have several web servers and therefore several access logs. The subtype element is appended to distinguish between logs of the same type by identifying the specific machine that is generating the log events. The subtype can include up to three levels in itself - this can be useful when formulating your subtypes. For example, we could end up with a complete tag like web.apache.error.pro.devo.server1 that identifies events from an Apache Web Server error log generated by the machine pro.devo.server1.
The main rules for the structure of tags are:
Each tag can have up to six levels of detail, the first three being the technology, brand, and type. The subtype can then contain up to three parts.
The tag can only be made of alphanumeric characters and periods. They cannot contain special characters or symbols.
The maximum length of a tag is 50 characters.
Correct Syntax | Incorrect Syntax |
|---|---|
box.unix web.apache web.apache.error.pro.devo.server1 a.b.c | box.unix. is incorrect because it ends with a period. web.apache.error.pro.devo/inc.server1 is incorrect because it contains a special character ( "/" ). a[ ].b.c is also incorrect because it contains special characters ("[]"). |
You can only use the tags in the List of Devo parsers or tags that have been defined for you by the Devo Professional Services team. These are the tags associated with the utility necessary to parse the associated events. In most cases, unless otherwise indicated, the tags are listed by the first few "fixed" levels, and subsequent levels can be user-defined, or "free".
Fixed tag elements
Some elements in the structure of a tag need to match a set of fixed values set by Devo. This is always true for technology and brand, but is also sometimes the case for type and subtype. Tag elements that must use predefined values are called fixed elements.
For example, the third element in Apache log tags is type and identifies the type of log event. Its values are limited to the log types produced by Apache and recognized by Devo:
If the type is error (as in web.apache.error), Devo recognizes the events as Apache log error events and will parse it according to the Apache error format.
If the type is accessclf (as in web.apache.accessclf), Devo recognizes the events as generated by an Apache access log and will parse it according to the corresponding Apache log format.
Free tag elements
In some cases, the type and subtype elements can be either partly or completely user-defined. Let's look at the example of Apache Web Server logs again. The technology and brand must be web and apache. The type element must have a recognized value from among the log type generated by the web server. The subtype element, however, is partly free because the user can define the values but Devo requires there to be three parts and that they conform to a guide that will help to further identify the type of data. Check out the web.apache tag article for more details.
The unknown unknowns
When Devo does not recognize the event's tag or when the event is received without a tag, Devo saves it with an eventdate in the unknown.unknown table like this:
Proprietary logs or data from a product not yet supported by Devo can be marked with the tag my.app. This tag has been defined specifically for all types of common data coming from unknown sources. The first time Devo receives an event with a tag starting with my.app, it will generate a notification to inform you.
