Table of Contents | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
About complex types
In Devo, complex data types provide flexible and powerful structures for querying and manipulatingdata logs. These types include set, map, array, and tuple, each with unique properties allowing users to efficiently handle diverse data operations. This article outlines the key characteristics and operations associated with each data type.
Type | Description | Example |
---|---|---|
Unordered, unique collection of values |
| |
Key-value pairs, unique keys |
| |
Ordered collection, allows duplicates |
| |
Ordered, immutable collection, mixed types |
| |
Key-value pairs, unordered, unique keys, mixed types |
|
...
Info |
---|
The JSON type is not considered a complex type but it is included in this documentation for similarity purposes. Currently, it is not possible to cast JSON natively from/to a complex type. |
...
Set
A set in Devo represents an unordered collection of unique values, meaning no duplicates are allowed. The order of elements in a set is not guaranteed, making it ideal for operations where uniqueness is more important than sequence.
Example:
Code Block |
---|
{1, 2, 3} |
Key Characteristics:
Unordered: Elements do not follow a specific order.
Unique: No duplicates are allowed.
Related operations:
Operation | Description | Syntax | Example | Result Value | Result Type |
---|---|---|---|---|---|
creates a set from elements | mkset (ele_1, ... ,ele_n) | mkset (1.33,2,3,4) | {1.33, 2, 3, 4} | set(float) | |
Checks if a set is empty | isempty (set) | isempty ({1, 2, 3}) | false | bool | |
Returns the length of a set | length (set) | length ({1, 20, 3, 4, 20, 20}) | 4 | int | |
Checks the occurrence of | has (set, ele) | has ({31, 2, 77}, 77) | true | bool | |
Checks the occurrence of | `in` (ele, set) | `in` (77, {30, 2, 77}) | true | bool | |
Checks whether a set | has (set_1, set_2) | has ({1, 2 , 3, 4}, {2, 1}) | true | bool | |
Checks whether a set | `in` (set_1, set_2) | `in` ({1, 2 , 3, 4}, {2, 1}) | false | bool | |
Adds some value to a set | add (set, ele) | add ({30, "Peter", 77}, “Ann") | {30, Peter, 77, Ann} | set(str) | |
Union of two sets | add (set_1, set_2) | add ({5, "Peter", 77}, {77, "Ann", 400}) | {5, Peter, 77, Ann, 400] | set(str) | |
Joins the strings of a given | join (set_str) | join ({1, "Peter", 77, 77, "Ann"}) | 1Ann77Peter | str | |
Intersection of two sets | band (set_1, set_2) | band ({30, 2, 77}, {77, 400}) | {77} | set(int) | |
Deletes a value from a set | sub (set, ele) | sub ({30, 2, 77}, 400) | {30, 2, 77} | •set(int) | |
Difference of two sets | sub (set_1, set_2) | sub ({30, 2, 77}, {400, 30}) | {2, 77} | set(int) |
Common Operations:
Code Block |
---|
from siem.logtrust.web.activity |
Create a set from values:
Code Block select mkset(srcHost, params) as mkset_string
Convert an array to a set:
Code Block select set([1, 2, 2, 3]) as toset_int_1_2_3
Check if a set contains a specific value:
Code Block select toset_int -> 2 as has
Code Block select 2 in toset_int_1_2_3 as _in
Add values to a set:
Code Block select "new value" + mkset_string as add_va
Get distinct values in a set out of a grouped field
Code Block select join(mkset_string, ",") as join_set
Get distinct values in a set out of a grouped field
Code Block group select collectdistinct(responseLength) as responseLength_sizedistinct
Test them together in Data Search
Code Block |
---|
from siem.logtrust.web.activity //create a set select mkset(srcHost,params) as mkset_string select {srcHost, params} //convert an into a set select [1,2,3] as array, set(array) as toset_int, //check if the set is empty isempty(mkset_string) as isemty, //what is the length of the array length(mkset_string) as length, //does the set contain a specific item? toset_int -> 2 as has, //add values to a set "new value" + mkset_string as add_va, //join a set. Concatenates all the values of a set join(mkset_string, ",") as join_set //Get distinct values in a set out of a grouped field //group select collectdistinct(responseLength) as responseLength_sizedistinct |
...
Map
A map in Devo is a collection of key-value pairs where each key is unique, but values can be duplicated. Maps allow for efficient lookups by keys and are useful when working with datasets that need to associate properties or attributes to specific items.
...
Code Block |
---|
{"src": srcPort, "serverPort": serverPort} |
Key Characteristics:
Key-Value Structure: Each pair associates a key with a value.
Unique Keys: Each key must be unique, though values can repeat.
Related operations:
Operation | Description | Syntax | Example | Result Value | Result Type | ||
---|---|---|---|---|---|---|---|
Creates a map from | mkmap (key_1, val_1, ... , key_n, val_n) | mkmap (10, 1001, 20, 1.33) | {10:1001, 20:1.33} | map(int, float) | |||
collectdistinct() (aggr) | Creates a map of distinct values after an aggregation | collectdistinct(aggr_field) |
| {{1, 3, 20, 4}} | map(int, int,int,int) | ||
Checks if a map is empty | isempty (map) | isempty ({1:10, 2:20, 3:30}) | false | bool | |||
Returns the length of a map | length (map) | length ({1:"a", 2:"b", 3:"c"}) | 3 | int | |||
Checks the | has (map, key) | has ({1:"aaa", 2:"bbb", 77:"ccc"}, 77) | true | bool | |||
Checks the | `in` (key, map) | `in` (77, {1:"aaa", 2:"bbb", 77:"ccc"}) | true | bool | |||
Adds some key-value | add (map, key, value) | add ({5:“Peter", 10:“Ann"}, 4, “xxx") | {5=Peter, 10=Ann, 4=xxx} | map(int, str) | |||
Concatenation of two | add (map_1, map_2) | add ({5:"Peter", 10:"Ann"}, {4:"xxx"}) | {5=Peter, 10=Ann, 4=xxx} | map(int, str) | |||
Given a map, returns the | at (map, key) | at ({"Hello":77, 4:4000}, "Hello") | 77 | int | |||
Intersection of two maps : | band (map_1, map_2) | band ({5:"Five", 2:"Two"}, {2:"xxx"}) | {2=Two} | map(int, str) | |||
Restrict a map to a set of | band (map, set) | band ({5:"Five", 2:"Two", 6:"Six"}, {6, 2}) | {2=Two, 6=Six} | map(int, str) | |||
Deletes the key-value pair | sub (map, key) | sub ({5:"Five", 2:"Two", 6:"Six"}, 5) | {2:“Two", 6:“Six"} | map(int, str) | |||
Deletes the key-value pairs | sub (map, set) | sub ({5:"Five", 2:"Two", 6:"Six"}, {2, 5, 10}) | {6=Six} | map(int, str) | |||
Difference of two maps | sub (map_1, map_2) | sub ({5:"Five", 2:"Two", 6:"Six"} {5:“xxx"}) | {2:“Two", 6:“Six"} | map(int, str) | |||
Returns the set of keys of a given map | keys (map) | keys ({"Hello":77, 4:4000}) | {Hello, 4} | set(str) | |||
Returns the array of values of a given map | values (map) | values ({"Hello":77, 4:4000}) | [77, 4000] | array(int) |
Common Operations:
Create a map:
Code Block select mkmap("b",7,"c",6,"a",5) as map1
Code Block select mkmap("src", srcPort, "serverPort", serverPort) as map2
Check if a map is empty:
Code Block select isempty(map1) as is_empty
Get the length of a map (number of key-value pairs):
Code Block select length(map1) as map_length
Access a value by key:
Code Block select map1 -> "b" as src_value
Append new key-value pairs:
Code Block select map1 + map2 as map3
Subtract pairs
Code Block select map3 - map2 as _substract_pairs
Code Block select map3 - "b" as _substract_key_b
Return the value of a give key
Code Block select map3["b"] as _return_7
Return all keys or values from a map:
Code Block select keys(map3) as _keys_set
Code Block select values(map3) as _values_array
Test them together in Data Search
Code Block | ||
---|---|---|
| ||
from siem.logtrust.web.activity //create a map select mkmap("b",7,"c",6,"a",5) as map1 select {"src":srcPort, "serverPort": serverPort} as map2 //Checks if a map is empty select isempty(map1) as _false //Returns the length of a map select length(map1) as _length //check the occurrence of key "b"? select map1 -> "b" as _true //append of new pairs select map1 + map2 as map3 //subtract pairs select map3 - map2 as _substract_pairs select map3 - "b" as _substract_key_b //return the value of a give key select map3["b"] as _return_7 //return all the keys or values of a map select keys(map3) as _keys_set select values(map3) as _values_array |
...
Array
An array is an ordered collection of elements where duplicates are allowed. Arrays are especially useful for performing a range of operations such as sorting, summing, and filtering. Unlike sets, arrays preserve the order of elements, making them essential for ordered data manipulation.
...
Code Block |
---|
["usernameJohn", "usernameMary", "usernameJoe", "usernameJane"] |
Key Characteristics:
Ordered: Elements follow a specific sequence.
Allows Duplicates: Identical elements can appear multiple times.
Related operations:
Operation | Description | Syntax | Example | Result Value | Result Type |
---|---|---|---|---|---|
Creates an array from elements | mkarray (ele_1, … ,ele_n) | mkarray (1.33,2,3,4) mkarray (“One", "two", "Number 3“) [ “One", 77 ] mkarray (domain, responseTime) | [1.33, 2, 3, 4] | array(float) array(str) array(str) array(str) | |
Creates an array by splitting a string field | split (string) |
| [1, 2, 3, 4] | array | |
Checks if an array is empty | isempty (array) | isempty ([1, 2, 3]) isempty ([ ]) | false true | bool | |
Returns the length of an array | length (array) | length ([1, 2, 3, 4, 5, 6]) | 6 | Int | |
Checks the occurrence of a specified value in a given array | •has (array, ele) •array -> ele | has ([30, 2, 77], 77) [30, 2, 77] -> 77 | true true | bool | |
Checks the occurrence of a specified value in a given array | `in` (ele, array) ele <- array | `in` (77, [30, 2, 77]) 77 <- [30, 2, 77] | true true | bool | |
Appends some value to the end of an array | add (array, ele) | add ([30, "Peter", 77], "Ann") | [30, Peter, 77, Ann] | array(str) | |
Concatenates two arrays | add (array_1, array_2) | add ([5, "Peter", 77], [77, "Ann", 400]) | [5, Peter, 77, 77, Ann, 400] | array(str) | |
Joins the strings of a given | join (array_str) | join ([1, "Peter", 77, 77, "Ann"]) | 1Peter7777Ann | str | |
Returns the n-th element of | at (array, n) | at ([10, 20, 3.33, 40], 1) | 20 | float | |
Returns a subarray between | at (array, from, to) | at ([10, 20, 30, 40], 1, 3) | [20, 30] | array(int) | |
Returns the index of the first | indexof (array, ele) | indexof ([10, 20, 3.33, 40], 3.33) | 2 | int | |
Drops null elements of a given array | dropnulls (array) | dropnulls ([3, null, 77, null]) | [3, 77] | array(int) | |
Sorts the elements of a | sort (array) | sort ([3, 77, 30, 1) | [1, 3, 30, 77] | array(int) | |
Reverses the elements of | reverse | reverse ([3, 77, 30, 1) | [1, 30, 77, 3] | array(int) | |
Sums the values of a | sum (array) | sum([1,2,3,4]) | 10 | Int |
Common Operations:
Create an array:
Code Block select mkarray(username, ip4(srcHost), mm2coordinates(ip4(srcHost)), true) as array1 select ["hi", "i_am_in_a_set"] as array2
Convert a set to an array:
Code Block select {serverPort, srcPort} as set1
Code Block select array(set1) as array3
Check if an array contains a value:
Code Block select "hi" in array2
Code Block select array2 -> "hi"
Add a value to an array:
Code Block select array2 + "example" as array_with_example
Reverse an array:
Code Block select reverse(array2) as array_reversed
Code Block select reverse("hello") as _treat_strings_as_arrays
Drop null values from an array:
Code Block select dropnulls(array2) as array_without_nulls
Sum numeric arrays:
Code Block select sum([1,5,8]) as array_sum
length of an array
Code Block select length(array2)
Test them together in Data Search
Code Block |
---|
from siem.logtrust.web.activity //create an array select mkarray(username, ip4(srcHost), mm2coordinates(ip4(srcHost)), true) as array1 select ["hi", "i_am_in_a_set"] as array2 //convert a set to an array select {serverPort, srcPort} as set1 select array(set1) as array3 //filter or check the occurrance of a value in an array select "hi" in array2 select array2 -> "hi" //length of an array select length(array2) //add a value select array2 + "example" as array_with_example //drop nulls select dropnulls(array2) as array_without_nulls //reverse an array select reverse(array2) as array_reversed select reverse("hello") as _treat_strings_as_arrays //sum numeric arrays select sum([1,5,8]) as _14 |
...
Tuple
A tuple in Devo is an ordered collection of elements that can be of mixed types (e.g., strings, numbers, booleans). Unlike arrays, tuples maintain the individual types of each element and are immutable, meaning that once created, the elements cannot be changed.
...
Code Block |
---|
(1.33, "text", true) |
Key Characteristics:
Ordered: Elements are stored in a specific sequence.
Immutable: The values in a tuple cannot be modified after creation.
Allows Duplicates: Identical elements can appear more than once.
Allows Mixed Types: Elements can be of any type (e.g., string, integer, boolean).
Related operations:
Operation | Description | Syntax | Example | Result Value | Result Type |
---|---|---|---|---|---|
Creates a tuple with specified elements. |
|
|
|
| |
Tuple literal | Creates a tuple using parentheses. |
|
|
|
|
Access by index | Access tuple elements using index. Supports negative indexing. |
|
| First element Last element | Matches element type |
Access tuple elements by index using a function. |
|
| First element | Matches element type | |
Filter by type | Filters tuple elements based on data type or pattern. |
|
| Matches filtered criteria |
|
Filter by value | Filters tuple elements based on value comparison. |
|
| Matches filtered criteria |
|
Sub-query (key lookup) | Finds the occurrence of a key in another dataset during the same period. |
|
| Matched key occurrence | Matches key type |
Sub-query (field lookup) | Returns a field from another table matching a condition. |
|
|
| Matches field type |
Sub-query (tuple matching) | Returns a tuple from another table matching specified conditions. |
|
|
|
|
Common Operations:
Create a tuple:
Code Block select mktuple(username, ip4(srcHost), mm2coordinates(ip4(srcHost)), true) as tuple
Code Block select (username ,srcPort, ip4(srcHost), true) as tuple2
Access tuple elements by index:
Code Block select tuple[0] as first_item_from_tuple
Code Block select tuple[-1] as last_item_from_tuple_
Code Block select at(tuple,0) as first_item_from_tuple2
Filter tuples by type:
Code Block where tuple[0] -> "@"
Code Block where tuple[-1] not in (ip4(95.63.39.51))
SUB-QUERY: find the occurrence of a specific IP in another table during the same period of time
Code Block select (from siem.logtrust.web.navigation group by srcHost select srcHost) -> srcHost as _ip_occurrence_in_another_table
SUB-QUERY: return the "origin" field in another table matching by user email
Code Block select (from siem.logtrust.web.navigation group by userEmail, origin)[username] as userInSubq
SUB-QUERY: return the tuple (userEmail, count()) from another table matching by the tuple (email, level)
Code Block select (from siem.logtrust.web.navigation group by userEmail, level select userEmail, count())[username, level] as match
Test them together in Data Search
Code Block |
---|
from siem.logtrust.web.activity //create a tuple with multiple types select mktuple(username, ip4(srcHost), mm2coordinates(ip4(srcHost)), true) as tuple select (username ,srcPort, ip4(srcHost), true) as tuple2 //some ways to select the first item from a tuple select tuple[0] as first_item_from_tuple select tuple[-1] as last_item_from_tuple_ select at(tuple,0) as first_item_from_tuple2 //SUB-QUERY: find the occurrence of a specific IP in another table during the same period of time select (from siem.logtrust.web.navigation group by srcHost select srcHost) -> srcHost as _ip_occurrence_in_another_table //SUB-QUERY: return the "origin" field in another table matching by user email select (from siem.logtrust.web.navigation group by userEmail, origin)[username] as userInSubq //SUB-QUERY: return the tuple (userEmail, count()) from another table matching by the tuple (email, level) select (from siem.logtrust.web.navigation group by userEmail, level select userEmail, count())[username, level] as match //it is possible to filter each item by the underlying data type where tuple[0] -> "@" where tuple[-1] not in (ip4(95.63.39.51)) |
...
JSON
Then JSON format is fully supported in Devo. It is useful for representing structured data in an unordered format, enabling flexible data storage and manipulation.
...
Code Block |
---|
{ "a": "b", "c": 1, "d": [1, 2, 3], "e": { "f": 1 } } |
Key Characteristics:
Unordered: The sequence of keys is not guaranteed.
Unique Keys: Each key must be unique within the object.
Mixed Types: Values can be strings, numbers, booleans, arrays, or other JSON objects.
Flexible: Supports nested objects and arrays for complex structures.
...
Operation | Description | Syntax | Example | Result | Result Type |
---|---|---|---|---|---|
Extracts values from JSON objects using a |
|
| Extracted value from JSON | Matches field type | |
Compiles a string as a |
|
| A compiled |
| |
Returns the data type of a value inside a JSON object. |
|
|
|
| |
Converts a string field to a JSON data type. |
|
| JSON object |
| |
Converts a JSON object or other data type to a string. |
|
| Extracted value as a string |
| |
Given a map, returns the | at (json, key) |
| Extract a value from a key | Matches field type | |
Converts a JSON object (json data type) into string data type. | stringify(json_type) | stringify(json_type) | {“a“: 1, “b“: ”c”} |
|
Common Operations:
Parse String to JSON
Code Block select jsonparse("{\"str\": \"hi\", \"int\": 1}") as json
Extract a Property Value
Code Block select json["int"] as extract_int //doesn't work in Data Search at the moment
Code Block select jqeval(jqcompile(".int"), json) as extract_int_alt
Convert a JSON to a string
Code Block select stringify(json) as json_to_strinc
Determine Data Type of a JSON Value
Code Block select label(jqeval(jqcompile(".int"), json)) as int_type
Test them together in Data Search
Code Block |
---|
from siem.logtrust.web.activity //create a JSON object select jsonparse("{\"str\": \"hi\", \"int\": 1}") as json //Extract a Property Value //select json["int"] as extract_int //doesn't work in Data Search at the moment select jqeval(jqcompile(".int"), json) as extract_int_alt //Convert a JSON to a string select stringify(json) as json_to_strinc //Determine Data Type of a JSON Value select label(jqeval(jqcompile(".int"), json)) as int_type |