Complex data types

Set

Unordered, unique collection of values

{1, 2, 3}

Map

Key-value pairs, unique keys

{"a": 1, "b": 2}

Array

Ordered collection, allows duplicates

["item1", "item2", "item3"]

Tuple

Ordered, immutable collection, mixed types

(1.33, "text", true)

JSON

Key-value pairs, unordered, unique keys, mixed types

{"a": "b", "c": 1, "d" : [1,2,3], "e": {"f": 1}}

mkset or {}

creates a set from elements

mkset (ele_1, ... ,ele_n)
{ele_1, ... , ele_n}

mkset (1.33,2,3,4)
mkset ("One", "two", "Number 3")
mkset (domain, responseTime)
mkset (responseTime, srcPort)

{1.33, 2, 3, 4}
{One, two, Number 3}
{One, 77}
{self, 12}
{12, 37406}

set(float)
set(str)
set(str)
set(str)
set(int)

isempty

Checks if a set is empty

isempty (set)

isempty ({1, 2, 3})
isempty ({ })

false
true

bool

length

Returns the length of a set

length (set)

length ({1, 20, 3, 4, 20, 20})

4

int

has (->)

Checks the occurrence of
a specified value in a
given set

has (set, ele)
set -> ele

has ({31, 2, 77}, 77)
{31, 2, 77} -> 77

true
true

bool

in (<-)

Checks the occurrence of
a specified value in a
given set

`in` (ele, set)
ele <- set

`in` (77, {30, 2, 77})
 77 <- {30, 2, 77}

true
true

bool

has (->)

Checks whether a set
contains another set

has (set_1, set_2)
set_1 -> set_2

 has ({1, 2 , 3, 4}, {2, 1})
 {1, 2 , 3, 4} -> {2, 1}

true
true

bool

in (<-)

Checks whether a set
contains another set

`in` (set_1, set_2)
set_1 <- set_2

`in` ({1, 2 , 3, 4}, {2, 1})
 {1, 2 , 3, 4} <- {2, 1}

false
false

bool

add (+)

Adds some value to a set

add (set, ele)
add (ele, set)

add ({30, "Peter", 77}, “Ann")
 {30, "Peter", 77} + “Ann"

{30, Peter, 77, Ann}
{30, Peter, 77, Ann}

set(str)
set(str)

add (+)

Union of two sets

add (set_1, set_2)
 set_1 + set_2

add ({5, "Peter", 77}, {77, "Ann", 400})
 {5, "Peter", 77} + {77, "Ann", 400}

{5, Peter, 77, Ann, 400]
{5, Peter, 77, Ann, 400]

set(str)
set(str)

join *

Joins the strings of a given
set in a single string
(separataror can be added)
NOTE : no order granted

join (set_str)
join (set_str, sep)

join ({1, "Peter", 77, 77, "Ann"})
join ({1, "Peter", 77, 77, "Ann"} , "_x_")

1Ann77Peter
1_x_Ann_x_77_x_Peter

str

band

Intersection of two sets

band (set_1, set_2)
 set_1 & set_2

band ({30, 2, 77}, {77, 400})
band ({30, 2, 77}, {500, 400})
 {30, 2, 77} & {77, 400}
 {30, 2, 77} & {500, 400}

{77}
{}
{77}
{}

set(int)
set(int)
set(int)
set(int)

sub (-)

Deletes a value from a set

sub (set, ele)
set - ele

sub ({30, 2, 77}, 400)
 {30, 2, 77} - 400
sub ({30, "Peter", 77}, "Peter")
 {30, "Peter", 77} - "Peter"

{30, 2, 77}
{30, 2, 77}
{30, 77}
{30, 77}

•set(int)
•set(int)
•set(str)
•set(str)

sub (-)

Difference of two sets

sub (set_1, set_2)
 set_1 – set_2

sub ({30, 2, 77}, {400, 30})
 {30, 2, 77} - 400
sub ({30, "Peter", 77}, {"Peter", 30})
 {30, "Peter", 77} - {"Peter", 30}

{2, 77}
{2, 77}
{77}
{77}

set(int)
set(int)
set(str)
set(str)

mkmap or { : }

Creates a map from
keys and values

mkmap (key_1, val_1, ... , key_n, val_n)

mkmap (10, 1001, 20, 1.33)
mkmap (10, 1001, 20, "Yesterday")
mkmap ("One", 100, "Two", 200)
 { 1 : "Hello", 2 : "Bye" }

{10:1001, 20:1.33}
{10=1001, 20=Yesterday}
{One=100, Two=200}
{1=Hello, 2=Bye}

map(int, float)
map(int, str)
map(str, int)
map(int, str)

collectdistinct() (aggr)

Creates a map of distinct values after an aggregation

collectdistinct(aggr_field)

{{1, 3, 20, 4}}

map(int, int,int,int)

isempty

Checks if a map is empty

isempty (map)

isempty ({1:10, 2:20, 3:30})
isempty ({})

false
true

bool

length

Returns the length of a

map

length (map)

length ({1:"a", 2:"b", 3:"c"})

3

int

has (->)

Checks the
occurrence of a
specified key in a
given map

has (map, key)
map -> key

has ({1:"aaa", 2:"bbb", 77:"ccc"}, 77)
 {1:"aaa", 2:"bbb", 77:"ccc"} -> 77

true
true

bool

in (<-)

Checks the
occurrence of a
specified key in a
given map

`in` (key, map)
key <- map

`in` (77, {1:"aaa", 2:"bbb", 77:"ccc"})
 77 <- {1:"aaa", 2:"bbb", 77:"ccc"}

true
true

bool

add (+)

Adds some key-value
pair to a map

add (map, key, value)

add ({5:“Peter", 10:“Ann"}, 4, “xxx")

{5=Peter, 10=Ann, 4=xxx}

map(int, str)

add (+)

Concatenation of two
maps

add (map_1, map_2)
 map_1 + map_2

add ({5:"Peter", 10:"Ann"}, {4:"xxx"})
 {5:"Peter", 10:"Ann"} + {4:"xxx"}

{5=Peter, 10=Ann, 4=xxx}
{5=Peter, 10=Ann, 4=xxx}

map(int, str)
map(int, str)

at [ ]

Given a map, returns the
value associated to a
given key

at (map, key)
 map [key]

at ({"Hello":77, 4:4000}, "Hello")
 {"Hello":77, 4:4000} ["Hello"]
at ({1:"One", 2:"Two"}, 888)
 {1:"One", 2:"Two"} [888]

77
77
null
null

int
int
str
str

band

Intersection of two maps :
•Non-commutative
•Intersection by key
•Returns first map value

band (map_1, map_2)
 map_1 & map_2

band ({5:"Five", 2:"Two"}, {2:"xxx"})
 {5:"Five", 2:"Two"} & {2:"xxx"}
band ({5:"Five", 2:"Two"}, {6:“Six"})
 {5:"Five", 2:"Two"} & {6:“Six"}

{2=Two}
{2=Two}
{}
{}

map(int, str)
map(int, str)
map(int, str)
map(int, str)

band

Restrict a map to a set of
keys

band (map, set)
 map & set

band ({5:"Five", 2:"Two", 6:"Six"}, {6, 2})
 {5:"Five", 2:"Two", 6:"Six"} & {6, 2}
band ({5:"Five", 2:"Two"}, {10, 20})
 {5:"Five", 2:"Two"} & {10, 20}

{2=Two, 6=Six}
{2=Two, 6=Six}
{}
{}

map(int, str)
map(int, str)
map(int, str)
map(int, str)

sub (-)

Deletes the key-value pair
defined by a key

sub (map, key)
 map - key

sub ({5:"Five", 2:"Two", 6:"Six"}, 5)
 {5:"Five", 2:"Two", 6:"Six"} - 5}

{2:“Two", 6:“Six"}
{2:“Two", 6:“Six"}

map(int, str)
map(int, str)

sub (-)

Deletes the key-value pairs
defined by a set of keys

sub (map, set)
 map – set

sub ({5:"Five", 2:"Two", 6:"Six"}, {2, 5, 10})
 {5:"Five", 2:"Two", 6:"Six"} - {2, 5, 10}

{6=Six}
{6=Six}

map(int, str)
map(int, str)

sub (-)

Difference of two maps
(difference by key)

sub (map_1, map_2)
 map_2 – map_2

sub ({5:"Five", 2:"Two", 6:"Six"} {5:“xxx"})
 {5:"Five", 2:"Two", 6:"Six"}- {5:“xxx"}

{2:“Two", 6:“Six"}
{2:“Two", 6:“Six"}

map(int, str)
map(int, str)

keys

Returns the set of keys of a given map

keys (map)

keys ({"Hello":77, 4:4000})
keys ({99:"Monday", 4:4000})

{Hello, 4}
{99, 4}

set(str)
set(int)

values

Returns the array of values of a given map
NOTE : no order granted

values (map)

values ({"Hello":77, 4:4000})
values ({99:"Monday", 4:4000})

[77, 4000]
[4000, Monday]

array(int)
array(str)

mkarray or []

Creates an array from elements

mkarray (ele_1, … ,ele_n)
[ele_1, … , ele_n]

mkarray (1.33,2,3,4) mkarray (“One", "two", "Number 3“) 

[ “One", 77 ]

mkarray (domain, responseTime)

[1.33, 2, 3, 4]
[One, two, Number 3]
[One, 77] [self, 12]

array(float) array(str) array(str) array(str)

split

Creates an array by splitting a string field

split (string, string)

split("1.2.3.4", ".")

[1, 2, 3, 4]

array

isempty

Checks if an array is empty

isempty (array)

isempty ([1, 2, 3]) isempty ([ ])

false true

bool

length

Returns the length of an array

length (array)

length ([1, 2, 3, 4, 5, 6])

6

Int

has (->)

Checks the occurrence of a specified value in a given array

•has (array, ele) •array -> ele

has ([30, 2, 77], 77)

[30, 2, 77] -> 77

true

true

bool

in (<-)

Checks the occurrence of a specified value in a given array

`in` (ele, array) ele <- array

`in` (77, [30, 2, 77]) 

77 <- [30, 2, 77]

true

true

bool

add (+)

Appends some value to the end of an array

add (array, ele)
array + ele

add ([30, "Peter", 77], "Ann")
 [30, "Peter", 77] + "Ann"

[30, Peter, 77, Ann]
[30, Peter, 77, Ann]

array(str)
array(str)

add (+)

Concatenates two arrays

add (array_1, array_2)
array_1 + array_2

add ([5, "Peter", 77], [77, "Ann", 400])
[5, "Peter", 77] + [77, "Ann", 400]

[5, Peter, 77, 77, Ann, 400]
[5, Peter, 77, 77, Ann, 400]

array(str)
array(str)

join

Joins the strings of a given
array in a single string
(separataror can be added)

join (array_str)
join (array_str, sep)

join ([1, "Peter", 77, 77, "Ann"])
join ([1, "Peter", 77, 77, "Ann"], “_x_”)

1Peter7777Ann
1_x_Peter_x_77_x_77_x_Ann

str

at or [][]

Returns the n-th element of
an array

at (array, n)
array [n]

at ([10, 20, 3.33, 40], 1)
[10, 20, 3.33, 40] [2]
at ([10, 20, 3.33, "HELLO"], 3)
at ([10, 20, 333, 40], 777)

20
3.33
HELLO
null

float
float
str
int

at or [][]

Returns a subarray between
two indices

at (array, from, to)
array [from, to]

at ([10, 20, 30, 40], 1, 3)
 ["A", "B", "C", "D"] [1, 3]

[20, 30]
["B", "C"]

array(int)
array(str)

indexof

Returns the index of the first
appearance of a value in
an array (-1 if not present)
starting at 0

indexof (array, ele)

indexof ([10, 20, 3.33, 40], 3.33)
indexof ([10, 20, 333, 40], 777)
indexof (["Hello", "my friend"],"Hello")

2
-1
0

int

dropnulls

Drops null elements of a given array

dropnulls (array)

dropnulls ([3, null, 77, null])
dropnulls ([3, null, "HELLO", null])

[3, 77]
[3, HELLO]

array(int)
array(str)

sort

Sorts the elements of a
given array

sort (array)

sort ([3, 77, 30, 1)
sort (["AAA", "aaa", "44", 5])

[1, 3, 30, 77]
[44, 5, AAA, aaa]

array(int)
array(str)

reverse

Reverses the elements of
a given array

reverse
(array)

reverse ([3, 77, 30, 1)
reverse (["AAA", "aaa", "44", 5])

[1, 30, 77, 3]
[5, 44, aaa, AAA]

array(int)
array(str)

sum

Sums the values of a
numeric array

sum (array)

sum([1,2,3,4])
sum([1,2,3.5,4.5])

10
11

Int
float

mktuple

Creates a tuple with specified elements.

mktuple(elem_1, elem_2, ..., elem_n)

mktuple(username, ip4(srcHost), mm2coordinates(ip4(srcHost)), true)

(username, ip4, coordinates, true)

tuple(varied types)

Tuple literal

Creates a tuple using parentheses.

(elem_1, elem_2, ..., elem_n)

(username, srcPort, ip4(srcHost), true)

(username, srcPort, ip4, true)

tuple(varied types)

Access by index

Access tuple elements using index. Supports negative indexing.

tuple[index]

tuple[0]

tuple[-1]

First element

Last element

Matches element type

at()

Access tuple elements by index using a function.

at(tuple, index)

at(tuple, 0)

First element

Matches element type

Filter by type

Filters tuple elements based on data type or pattern.

where tuple[index] -> type

where tuple[0] -> "@"

Matches filtered criteria

bool

Filter by value

Filters tuple elements based on value comparison.

where tuple[index] in/not in (values)

where tuple[-1] not in (ip4(95.63.39.51))

Matches filtered criteria

bool

Sub-query (key lookup)

Finds the occurrence of a key in another dataset during the same period.

(from dataset group by key select key) -> key

(from siem.logtrust.web.navigation group by srcHost select srcHost) -> srcHost

Matched key occurrence

Matches key type

Sub-query (field lookup)

Returns a field from another table matching a condition.

(from dataset group by field1, field2)[key]

(from siem.logtrust.web.navigation group by userEmail, origin)[username]

origin field value

Matches field type

Sub-query (tuple matching)

Returns a tuple from another table matching specified conditions.

(from dataset group by tuple_field select fields)[key1, key2]

(from siem.logtrust.web.navigation group by userEmail, level select userEmail, count())[username, level]

(userEmail, count)

tuple(varied types)

jqeval()

Extracts values from JSON objects using a jq filter.

jqeval(jq_filter, json_field)

jqeval(jqcompile(".email"), json)

Extracted value from JSON

Matches field type

jqcompile()

Compiles a string as a jq filter to be used in jqeval.

jqcompile(filter)

jqcompile(".email")

A compiled jq filter

jq filter

label()

Returns the data type of a value inside a JSON object.

label(json_field)

label(json.email)

"string"

string

jsonparse()

Converts a string field to a JSON data type.

jsonparse(string_field)

jsonparse(data_string)

JSON object

json

str()

Converts a JSON object or other data type to a string.

str(value)

str(jqeval(jqcompile(".email"), json))

Extracted value as a string

string

at() []

Given a map, returns the
value associated to a
given key

at (json, key)
 json [key]

json["int"]

at(json, "int")

Extract a value from a key

Matches field type

stringify()

Converts a JSON object (json data type) into string data type.

stringify(json_type)

stringify(json_type)

{“a“: 1, “b“: ”c”}

string