Page Comparison

...

`unseparated_json_processor`	This is if the events come in json in one massive object use this.
`split_or_unseparated_processor`	This will determine if the log is split by `\n` or not.
`aws_access_logs_processor`	For AWS access logs and \n splits.
`single_json_object_processor`	This is for one JSON object.
`separated_json_processor`	Similar to other separators.
`bluecoat_processor`	For Bluecoat recipe.
`json_object_to_linesplit_processor`	Split by configured value.
`json_array_processor`	For JSON array processors
`json_line_arrays_processor`	Similar to other separators
`jamf_processor`	Jamf log processing.
`parquet_processor`	Parquet encoding.
`guardduty_processor`	For GuardDuty processors.
`vpc_flow_processor`	VPC service processor.
`alt_vpc_flow_processor`	VPC service processor.
`kolide_processor`	For Kolide service.
`json_array_vpc_processor`	VPC service processor.
`rds_processor`	RDS processor for the RDS service `unseparated_json_processor`. Use this if the events come in one massive JSON object.

Tagging

Tagging can be done in many different ways. One way tagging works is by using the file field definitions:

Code Block
"file_field_definitions": { "log_type": [ { "operator": "split", "on": "/", "element": 2 } ] },

These are the elements of the filename object:

...

If you look at the highlighted object filename, you can see that we are splitting and looking for the 2nd value. This starts at 0 like arrays. So:

0 = cequence-data
1 = cequence-devo-6x-NAieMI
2 = detector

"routing_template": "my.app.test_cequence.[file-log_type]"

Our final tag is my.app.test_cequence.detector

Options for filtering

...

direct_mode

...

Allowed values are true or false (default value is false). Set to true if the logs are sent directly to the queue without using s3.

...

file_field_definitions

...

Defined as a dictionary mapping of variable names (you decide) that lists parsing rules.

Each parsing rule has an operator with its own keys. Parsing rules are applied in the order they are listed in the configuration.

The split operator uses the on and element keys. The file name will split into pieces considering the character or character sequence specified in the on key, and will extract whatever it is at the specified element index, as in the example below.
The replace operator uses the to_replace and replace_with keys.

For example, if your filename is server_logs/12409834/ff.gz, this configuration would store the log_type as serverlogs:

Code Block
"file_field_definitions": { "log_type": [{"operator": "split", "on": "/", "element": 0}, {"operator": "replace", "to_replace": "_", "replace_with": ""}] }

...

filename_filter_rules

...

A list of rules to filter out entire files.

...

encoding

...

Takes any string. List of most common to least common: gzip, none, parquet, latin-1

...

ack_messages

...

Decide whether or not to delete messages from the queue after processing. It takes boolean values. If not specified, default is true. We recommend leaving this out of the config. If you see it in there, pay close attention to if it’s on or off.

...

file_format

type - A string specifying which processor to use.

...

single_json_object - Logs are stored as/in a JSON object.

single_json_object_processor config options:

key -(string) The key of where the list of logs is stored.

Code Block
config: {"key": "log"} fileobj: {..."log": {...}}

...

unseparated_json_processor - Logs are stored as/in JSON objects, which are written in a text file with no separator.

unseparated_json config options:

key - (string) where the log is stored
include (dict: maps names of keys outside of inner part to be included, which can be renamed).

If there is no key, that is, the whole JSON object is the desired log, set "flat": true

See aws_config_collector for example:

Code Block
fileobj: {...}{...}{...}

...

text_file_processor - logs are stored as text files, potentially with lines and fields separated with e.g. commas and newlines

text_file config options: includes options for how lines and records are separated (e.g. newline, tab, comma), good for csv style data.

...

line_split_processor –- logs stored in a newline separated file, works more quickly than separated_json_processor

config options: “json”: true or false. If setting json to true, assumes that logs are newline-separated json, and allows them to be parsed by the collector therefore enabling record-field mapping

...

separated_json_processor – logs stored as many json objects that have some kind of separator

config options: specify the separator e.g. “separator”: “||”. the default is newline if left unused.

Code Block
fileobj: {...}\|\|{...}\|\|{...}

...

jamf_processor – special processor for JAMF logs

...

aws_access_logs_processor – special processor for AWS access logs

...

windows_security_processor – special processor for Windows Security logs

...

vpc_flow_processor – special processor for VPC Flow logs

...

json_line_arrays_processor – processor for unseparated json objects that are on multiple lines of a single file.

Code Block
fileobj: {...}{...} {...}{...}{...} {...}

...

dict_processor – processor for logs that comes as python dictionary objects, i.e. in direct mode

...

config - a dictionary of information the specified file_format processor needs

...

record_field_mapping

...

a dictionary -- each key defines a variable that can be parsed out from each record (which may be referenced later in filtering)
e.g., we may want to parse something and call it "type", by getting "type" from a certain key in the record (which may be multiple layers deep).

Code Block
{"type": {"keys": ["file", "type"]}, "operations": [] }

keys is a list of how key values in the record to look into to find the value, its to handle nesting (essentially defining a path through the data). Suppose we have logs that look like this:

Code Block
{“file”: {“type”: { “log_type” : 100}}}

so if we want to get the log_type, we should list all the keys needed to parse through the json in order:

Code Block
keys: [“file”, “type”, “log_type”]

In many cases you will probably only need one key.

e.g. in flat json that isn’t nested

Code Block
{“log_type”: 100, “other_info”: “blah” ….}

here you would just specify keys: [“log_type”]. A few operations are supported that can be used to further alter the parsed information (like split and replace). This snippet would grab whatever is located at log[“file”][“type”] and name it as “type”. record_field_mapping defines variables by taking them from logs, and these variables can then be used for filtering. Let’s say you have a log in json format like this which will be set to devo:

Code Block
{“file”: {“value”: 0, “type”: “security_log”}}

Specifying “type” in the record_field_mapping will allow the collector to extract that value, “security_log” and save it as type. Now let’s say you want to change the tag dynamically based on that value. You could change the routing_template to something like my.app.datasource.[record-type]. In the case of the log above, it would be sent to my.app.datasource.security_log. Now let’s say you want to filter out (not send) any records which have the type security_log. You could write a line_filter_rule as follows:

{"source": "record", "key": "type", "type": "match", "value": "security_log" } We specified the source as record because we want to use a variable from the record_field_mapping. We specified the key as “type” because that is the name of the variable we defined. We specify type as “match” because any record matching this rule we want to filter out. And we specify the value as security_log because we specifically do not want to send any records with the type equalling “security_log” The split operation is the same as if you ran the python split function on a string.

Let’s say you have a filename “logs/account_id/folder_name/filename” and you want to save the account_id as a variable to use for tag routing or filtering.

You could write a file_field_definition like this:

"account_id": [{"operator": "split", "on": "/", "element": 1}]

This would store a variable called account_id by taking the entire filename and splitting it into pieces based on where it finds backslashes, then take the element as position one. In Python it would look like:

Code Block
filename.split(“/”)[1]

...

routing_template

...

More on processors:

file_format

type - A string specifying which processor to use.

single_json_object - Logs are stored as/in a JSON object.

single_json_object_processor config options:

key -(string) The key of where the list of logs is stored.

Code Block
config: {"key": "log"} fileobj: {..."log": {...}}

unseparated_json_processor - Logs are stored as/in JSON objects, which are written in a text file with no separator.

unseparated_json config options:

key - (string) where the log is stored
include (dict: maps names of keys outside of inner part to be included, which can be renamed).

If there is no key, that is, the whole JSON object is the desired log, set "flat": true

See aws_config_collector for example:

Code Block
fileobj: {...}{...}{...}

text_file_processor - logs are stored as text files, potentially with lines and fields separated with e.g. commas and newlines

text_file config options: includes options for how lines and records are separated (e.g. newline, tab, comma), good for csv style data.

line_split_processor –- logs stored in a newline separated file, works more quickly than separated_json_processor

config options: “json”: true or false. If setting json to true, assumes that logs are newline-separated json, and allows them to be parsed by the collector therefore enabling record-field mapping

separated_json_processor – logs stored as many json objects that have some kind of separator

config options: specify the separator e.g. “separator”: “||”. the default is newline if left unused.

Code Block
fileobj: {...}\|\|{...}\|\|{...}

jamf_processor – special processor for JAMF logs

aws_access_logs_processor – special processor for AWS access logs

windows_security_processor – special processor for Windows Security logs

vpc_flow_processor – special processor for VPC Flow logs

json_line_arrays_processor – processor for unseparated json objects that are on multiple lines of a single file.

Code Block
fileobj: {...}{...} {...}{...}{...} {...}

dict_processor – processor for logs that comes as python dictionary objects, i.e. in direct mode

config - a dictionary of information the specified file_format processor needs

record_field_mapping

A dictionary where each key defines a variable that can be parsed out from each record (which may be referenced later in filtering).

For example, we may want to parse something and call it type by getting type from a certain key in the record (which may be multiple layers deep).

Code Block
{"type": {"keys": ["file", "type"]}, "operations": [] }

The keys are a list of how to find a value and handle nesting (essentially, defining a path through the data).

Suppose we have logs that look like this:

Code Block
{“file”: {“type”: { “log_type” : 100}}}

If we want to get the log_type, we should list all the keys needed to parse through the JSON in order:

Code Block
keys: [“file”, “type”, “log_type”]

In many cases, you will probably only need one key, for example, in a flat JSON that isn’t nested:

Code Block
{“log_type”: 100, “other_info”: “blah” ….}

Here you would just specify keys: ["log_type"]. There are some operations that can be used to further alter the parsed information (like split and replace).

This snippet would grab whatever is located at log["file"]["type"] and name it as type. record_field_mapping defines variables by taking them from logs, and these variables can then be used for filtering.

Let’s say you have a log in JSON format like this which will be set to Devo:

Code Block
{“file”: {“value”: 0, “type”: “security_log”}}

Specifying type in record_field_mapping will allow the collector to extract the security_logand save it as type.

Now let’s say you want to change the tag dynamically based on that value. You could change the routing_template to something like my.app.datasource.[record-type]. In the case of the log above, it would be sent to my.app.datasource.security_log.

Now let’s say you want to filter out (not send) any records which have the type security_log. You could write a line_filter_rule as follows:

Code Block
{"source": "record", "key": "type", "type": "match", "value": "security_log" }

We specified the source as record because we want to use a variable from the record_field_mapping.
We specified the key as type because that is the name of the variable we defined.
We specified type as match because any record matching this rule we want to filter out.
And we specified the value as security_log because we specifically do not want to send any records with the type equalling security_log.

The split operation is the same as if you ran the Python split function on a string.

Let’s say you have a filename logs/account_id/folder_name/filename and you want to save the account_id as a variable to use for tag routing or filtering.

You could write a file_field_definition like this:

Code Block
"account_id": [{"operator": "split", "on": "/", "element": 1}]

This would store a variable called account_id by taking the entire filename and splitting it into pieces based on where it finds backslashes, then take the element as position one. In Python, it would look like this:

Code Block
filename.split(“/”)[1]

Tagging

Tagging can be done in many different ways. One way tagging works is by using the file field definitions:

Code Block
"file_field_definitions": { "log_type": [ { "operator": "split", "on": "/", "element": 2 } ] },

These are the elements of the filename object:

...

If you look at the highlighted object filename, you can see that we are splitting and looking for the 2nd value. This starts at 0 like arrays. So:

0 = cequence-data
1 = cequence-devo-6x-NAieMI
2 = detector

"routing_template": "my.app.test_cequence.[file-log_type]"

Our final tag is my.app.test_cequence.detector

Here is another example:

file_field_definitions

Defined as a dictionary mapping of variable names (you decide) that lists parsing rules.

Each parsing rule has an operator with its own keys. Parsing rules are applied in the order they are listed in the configuration.

The split operator uses the on and element keys. The file name will split into pieces considering the character or character sequence specified in the on key, and will extract whatever it is at the specified element index, as in the example below.
The replace operator uses the to_replace and replace_with keys.

For example, if your filename is server_logs/12409834/ff.gz, this configuration would store the log_type as serverlogs:

Code Block
"file_field_definitions": { "log_type": [{"operator": "split", "on": "/", "element": 0}, {"operator": "replace", "to_replace": "_", "replace_with": ""}] }

routing_template

A string defining how to build the tag to send each message, for example, my.app.wow.[record-type].[file-log_type]

" -- if the "type"

If the type extracted during record_field_mapping

were "

was null

"

, the record would be sent to the tag

"

my.app.wow.null

"

line_filter_rules

Options for filtering

Line-level filters

These are a list of

...

rules for filtering out

...

single events.

Expand

title	Example 1

We want to discard all the events that match these conditions:

Code Block

...

if record.eventName = "HeadObject" or record.eventName = "ListObjects" or record.eventName = "HeadBucket" or record.eventName = "GetBucketLocation" 
 do_not_send_record()

eventName is one of these values: HeadObject, ListObjects, HeadBucket, GetBucketLocation

In Devo, these criteria are specified with the next query. If everything is OK, after configuring the collector properly, there should not be any event if we run this query:

Code Block
from cloud.aws.cloudtrail.s3 where eventName = "HeadObject" or eventName = "ListObjects" or eventName = "HeadBucket" or eventName = "GetBucketLocation"

In this case, the key for the filter is the eventName, so first we need to add the key to the collector in the record_field_mapping section. After the record_field_mapping, we apply the corresponding filters in the line_filter_rules section. In this case, this would be as follows:

Code Block

"record_field_mapping": {
  "eventName": {
    "keys": ["eventName"]
  }
},
"line_filter_rules": [
    [{"source": "record", "key": "eventName", "type": "match", "value": "HeadObject"}],
    [{"source": "record", "key": "eventName", "type": "match", "value": "ListObjects"}],
    [{"source": "record", "key": "eventName", "type": "match", "value": "HeadBucket"}],
    [{"source": "record", "key": "eventName", "type": "match", "value": "GetBucketLocation"}]
]

Elements in different lists are OR conditions. Elements in the same list are AND conditions.

Note
Note that the logic for these filters is if they match the query, the collector won't send the event to Devo.

Expand

title	Example 2

What if we want to filter out the events that match this pseudocode query that has mixed conditions?

Code Block
if record.type != "ldap" OR (record.main-log_ornot == main-log AND record.type == "kube-api-server-audit"): do_not_send_record()

In this case, the keys for the filter are typeandmain-log_ornot, so first we need to add the keys to the collector in the record_field_mapping section. Once we’ve added the keys, we apply the corresponding filters. In this case, the filters would be as follows:

Code Block

"record_field_mapping": {
  "type": {
    "keys": ["type"]
  },
  "main-log_ornot": {
    "keys": ["main-log_ornot"]
  }
},
"line_filter_rules": [
	[{"source": "record", "key": "type", "type": "doesnotmatch", "value": "ldap"}],
    [
        {"source": "record", "key": "main-log_ornot", "type": "match", "value": "main-log"}, 
        {"source": "record", "key": "type", "type": "match", "value": "kube-apiserver-audit"}
    ]
]

Elements in different lists are OR conditions. Elements in the same list are AND conditions.

Note
Note that the logic for these filters is if they match the query, the collector won't send the event to Devo.

File-level filters

These are a list of rules to filter out entire files by the specified pattern applied over the file name.

Expand

title	Example 1

Code Block
"filename_filter_rules": [ [{"type": "match", "pattern": "CloudTrail-Digest"}], [{"type": "match", "pattern": "ConfigWritabilityCheckFile"}] ]

This will filter out files that contain CloudTrail-Digest or ConfigWritabilityCheckFile.

2024/01/01/CloudTrail-Digest-2024-01-01-00-00-00-123456789012.gz will be skipped.
2024/01/01/ConfigWritabilityCheckFile-2024-01-01-00-00-00-123456789012.gz will be skipped.

Expand

title	Example 2

Code Block
"filename_filter_rules": [ [{"type": "doesnotmatch", "pattern": "CloudTrail"}], [{"type": "match", "

...

pattern": "

...

CloudTrail-

...

Digest"}

...

This

...

will filter out files that do not contain CloudTrail or contain CloudTrail-Digest. For instance, files with a name like this:

2024/01/01/CloudTrail-2024-01-01-00-00-00-123456789012.gz will be processed.
2024/01/01/CloudTrail-Digest-2024-01-01-00-00-00-123456789012.gz will be skipped. Config can include "debug_mode": true to print out some useful information as logs come in.
For local testing, it is useful to set

...

ack_messages

...

to false

...

to try processing without eating from the queue. Be careful to remove this or set it to true when launching the collector. The default is to ack messages if it is not set.

If something seems wrong at launch, you can set the following in the collector parameters/job config

...

:

"debug

...

": true,
"do_not_send": true,
"ack_messages": false ← you will see duplicates if you turn this to false, just set to true when done.

This will print out data as it is being processed, stop messages from getting hacked, and at the last step,

...

data won’t send the data

...

. In this way, you can

...

easily check if something is

...

not working properly.

Versions Compared

Old Version 2

New Version Current

Key

Tagging

Options for filtering

Tagging

Options for filtering

Line-level filters

File-level filters