Custom Service Options for Custom SQS Collectors

file_format

type A string specifying which processor to use.

config A dictionary of the processor’s parameters.

record_field_mapping

A dictionary where each key defines a variable that can be parsed out from each record (which may be referenced later in filtering).

To parse something and call it type by getting type from a certain key in the record (which may be multiple layers deep).

Suppose we have logs that look like this:

To get the log_type, we should list all the keys needed to parse through the JSON in order:

In many cases, you will probably only need one key, for example, in a flat JSON that isn’t nested:

Here you would just specify keys: ["log_type"]. There are some operations that can be used to further alter the parsed information (like split and replace).

This snippet would grab whatever is located at log["file"]["type"] and name it as type. record_field_mapping defines variables by taking them from logs, and these variables can then be used for filtering.

If you have a log in JSON format like this which will be set to Devo:

Specifying type in record_field_mapping will allow the collector to extract the security_log and save it as type.

To change the tag using a field mapping, change the routing_template to something like my.app.datasource.[record-type]. In the case of the log above, it would be sent to my.app.datasource.security_log.

To filter out (not send) any records which have the type security_log, write a line_filter_rule as follows:

• We specified the source as record because we want to use a variable from the record_field_mapping.

• We specified the key as type because that is the name of the variable we defined.

• We specified type as match because any record matching this rule we want to filter out.

• And we specified the value as security_log because we specifically do not want to send any records with the type equalling security_log.

The split operation is the same as if you ran the Python split function on a string.

Given a filename logs/account_id/folder_name/filename and you want to save the account_id as a variable to use for tag routing or filtering.

You could write a file_field_definition like this:

This would store a variable called account_id by taking the entire filename and splitting it into pieces based on where it finds backslashes, then take the element as position one. In Python, it would look like this: