Parameter

Data type

Type

Value range / Format

Details

debug

bool

Discouraged

false / true

Do not include it. Enabling debug will incur additional costs.

id

int

Mandatory

Minimum length: 5
Maximum length: 5

Use a unique five digit number.

enabled

bool

Discouraged

false / true

If it is false the collector will have no input.

base_url

str

Mandatory

The URL of the SQS queue created during the authorization process.

aws_access_key_id

str

Discouraged

Any

Use cross account roles instead of keys.

aws_secret_access_key

str

Discouraged

Any

Use cross account roles instead of keys.

aws_base_account_role

str

Optional

Any

This is Devo's role. Remove it to use the default set by Devo.

aws_cross_account_role

str

Encouraged

Any

This is the role created during the authorization process.

aws_external_id

str

Encouraged

Any

This is the external ID created during the authorization process. If using a cross account role, the external ID is necessary.

ack_messages

bool

Optional

false / true

This must be true in production. Disabling it will cause duplicate ingestion, incurring a cost.

direct_mode

bool

Optional

false / true

Set to false for most scenarios.

do_not_send

bool

Discouraged

false / true

Set to true to not send the log to Devo.

sqs_visibility_timeout

int

Optional

Min: 120

Max: 43200

Specifies how long the message in the queue will be invisible to the collector after it is requested. If it is not processed and deleted within the allotted time in seconds, the message will be put back and may be processed again.

Set this parameter if the collector has to download large files and process them.

Should you reduce the timeout, you will need to wait for messages with the old timeout to become visible for the new timeout to take full effect.

sqs_wait_timeout

int

Discouraged

Min: 20

Max: 20

Time the collector waits to get a message. The default, 20 seconds, is recommended.

sqs_max_messages

int

Optional

Min: 1

Max: 1

Unused.

region

str

Optional

Example:

us-east-1

The AWS region, which must be the same as the region in the SQS URL.

compressed_events

bool

Optional

false / true

gzip decompression support. Select false unless the S3 objects are compressed with gzip.

The error ‘utf-8' codec can't decode byte 0xa9 in position 36561456: invalid start byte may indicate the events need to be decompressed.

encoding

str

Optional

Options from most used to least used.

• gzip

• none

• parquet

• latin-1

• Any Unicode encoding supported by python, such as ascii or utf-16.

split_or_unseparated_processor

Selects the processor by detecting \n. If unsure, this processor is recommended.

line_split_processor

Split an object into logs at each \n character using Python splitlines. Optionally, the log can be split using:

• chunks, a boolean parameter which enables the remaining parameters.

• indices, a Python integer array which selects lines from the object.

• substrings, a Python string array which selects lines from the object containing the configured strings.

• regex, a Python string array which selects lines from the object using python re.findall.

text_file_processor

Finds header information and adds it to each event. The line_separator parameter is required. The header can be identified using one of these parameters.

• header (boolean) and header_field_separator to get headers from the first line of data.

• field_names for a manual header.

• field_separator for numbered fields.

Additionally, if use_json is true, Python orjson will convert the data to JSON.

unseparated_json_processor

Split a JSON array into individual Devo events with Python raw_decode. The key parameter is permitted. The parameter may be a string or array. The include parameter is permitted. It a dictionary which renames JSON keys. The rename parameter can rename keys selected by the key parameter.

single_json_object_processor

For messages containing one JSON object. Uses Python orjson to process a single JSON object. Not for arrays. The key parameter is permitted.

separated_json_processor

Similar to other separators. The default separator is \n. The separator parameter is permitted.

json_object_to_linesplit_processor

Split by configured value. The key string parameter or keys array parameter are permitted.

unseparated_json_processor_extract_key

The unseparated_json_processor with an additional extraction_key parameter permitted. Use this when filtering on two levels of JSON keys.

If the log message has this format

{
  "id": 1,
  "timestamp": 2,
  "logEvents": {
    "another_id": 3,
    "another_timestamp": 4,
    "message": "send to devo"
  }
} 

The configuration

"file_format": {
  "type": "unseparated_json_processor_extract_key",
  "config": {
    "key": "logEvents",
    "extraction_key": "message"
  }
},

will send

send to devo

to Devo.

json_array_processor

Split a JSON array into individual Devo events with Python orjson. The key string parameter or keys array parameter are permitted.

json_line_arrays_processor

Processes JSON separated by \n. Use separated_json_processor instead.

aws_access_logs_processor

For AWS access logs and \n splits.

bluecoat_processor

Bluecoat.

jamf_processor

Jamf logs.

parquet_processor

Parquet processing using Python pandas.read_parquet. The data is converted to JSON.

guardduty_processor

For GuardDuty processors.

vpc_flow_processor

AWS VPC.

alt_vpc_flow_processor

Used for exception handling.

kolide_processor

For Kolide.

json_array_vpc_processor

AWS VPC.

rds_processor

RDS processor for the RDS service unseparated_json_processor. Use this if the events come in one massive JSON object.

windows_security_processor

Windows security logs.

file_format

type A string specifying which processor to use, from the list above.

config A dictionary of the processor’s parameters.

record_field_mapping

A dictionary where each key defines a variable that can be parsed out from each record (which may be referenced later in filtering).

To parse something and call it type by getting type from a certain key in the record (which may be multiple layers deep).

{"type": {"keys": ["file", "type"]},	"operations": []	}

Suppose we have logs that look like this:

{“file”: {“type”: { “log_type” : 100}}}

To get the log_type, we should list all the keys needed to parse through the JSON in order:

keys: [“file”, “type”, “log_type”]

In many cases, you will probably only need one key, for example, in a flat JSON that isn’t nested:

{“log_type”: 100, “other_info”: “blah” ….}

Here you would just specify keys: ["log_type"]. There are some operations that can be used to further alter the parsed information (like split and replace).

This snippet would grab whatever is located at log["file"]["type"] and name it as type. record_field_mapping defines variables by taking them from logs, and these variables can then be used for filtering.

If you have a log in JSON format like this which will be set to Devo:

{“file”: {“value”: 0, “type”: “security_log”}}

Specifying type in record_field_mapping will allow the collector to extract the security_log and save it as type.

To change the tag using a field mapping, change the routing_template to something like my.app.datasource.[record-type]. In the case of the log above, it would be sent to my.app.datasource.security_log.

To filter out (not send) any records which have the type security_log, write a line_filter_rule as follows:

{"source": "record", "key": "type", "type": "match", "value": "security_log" } 

• We specified the source as record because we want to use a variable from the record_field_mapping.

• We specified the key as type because that is the name of the variable we defined.

• We specified type as match because any record matching this rule we want to filter out.

• And we specified the value as security_log because we specifically do not want to send any records with the type equalling security_log.

The split operation is the same as if you ran the Python split function on a string.

Given a filename logs/account_id/folder_name/filename and you want to save the account_id as a variable to use for tag routing or filtering.

You could write a file_field_definition like this:

"account_id": [{"operator": "split", "on": "/", "element": 1}]

This would store a variable called account_id by taking the entire filename and splitting it into pieces based on where it finds backslashes, then take the element as position one. In Python, it would look like this:

filename.split(“/”)[1]