Contains tokens (toktains)

[ 1 Description ] [ 2 How does it work in the search window? ] [ 3 How does it work in LINQ? ]

Description

You can apply this operation either as a Filter or Create column operation:

Filter	Creates a filter that returns only those strings including a specific token. Optionally, you can add one or two boolean values to extend the left and right length of the token.
Create column	Adds a new Boolean column that shows true when a specific token is present in a given string. Optionally, you can add one or two boolean values to extend the left and right length of the token.

This operation is case sensitive. Use the Contains tokens - case insensitive (weaktoktains) operation if you need to apply this operation ignoring case.

What is a token?

A token is simply a unit of data composed of alphanumeric characters separated by ASCII symbols. Devo also recognizes as a token any value assigned the ip or ip6 data format. Therefore, not only will Devo identify 10.0.1.2 and aa:bb:cc:dd as tokens, but also their component parts, 10, 0, 1, 2, aa, bb, cc, and dd because these component parts are delimited by ASCII symbols (the periods and colons).

Learn more in How Devo indexes data.

How does it work in the search window?

Select Filter / Create column in the search window toolbar, then select the Contains tokens operation. This operation requires at least two arguments (Value and contains). Optionally, you can add the right extended and left extended arguments to expand the length of the token. Learn more about these arguments in the example below.

Argument	Data type	More information

Argument	Data type	More information
Value ^mandatory	string
contains ^mandatory	string
left extended	boolean	Add this argument if you want to get strings that contain the indicated token with alphanumerical values to the left. For example, if you enter 10 as a token and set the left extended argument to true, you will get strings containing, for example, -10- or .10. but also abc10- or 12310-
right extended	boolean	Add this argument if you want to get strings that contain the indicated token with alphanumerical values to the right. For example, if you enter 10 as a token and set the right extended argument to true, you will get strings containing, for example, -10- or .10. but also -10abc or -10123

If you use the Create column operation, the data type of the values in the new column is boolean (true or false).

Example

In the demo.ecommerce.data table, we want to get only the events that include the token 09 in the uri column strings (value 09 delimited by ASCII symbols). In addition, we also want to get those strings that contain alphanumerical values to the left of the value 09, so we will extend the length of the token using the left extended argument. To do it, we will apply a Filter using the Contains tokens operation.

The arguments needed for the filter are:

Value - uri column
contains - Click the pencil icon and enter 09
left extended - Click the pencil icon and enter true
right extended - Click the pencil icon and enter false

Click Filter data and you will see the following result:

The filter looks for values including the token 09, that is to say, the value 09 delimited by ASCII symbols (periods, colons...). However, adding the left extended and right extended arguments makes the filter include additional strings:

Since the left extended argument was set to true, rows with uri values containing alphanumeric values to the left of the token are included.
Since the right extended argument was set to false, rows with uri values containing alphanumeric values to the right of the token are filtered out.

Click Create column and follow the same steps to add a new Boolean column that shows true when the conditions explained above are met.

How does it work in LINQ?

Use the operator where... to apply the Filter operation and select... as... to apply the Create column operation. These are the valid formats of the Contains tokens operation:

toktains(string1, string2)
toktains(string1, string2, boolean_left)
toktains(string1, string2, boolean_left, boolean_right)

Examples

You can copy the following LINQ script and try the above example on the demo.ecommerce.data table:

from demo.ecommerce.data
  where toktains(uri, "09", true, false)

And this is the same example using the Create column operation:

from demo.ecommerce.data
  select toktains(uri, "09", true, false) as token_09