Custom SQS Collector

[ 1 Purpose ] [ 2 Authorize It ] [ 3 Gather Information ] [ 4 Run It ]

Purpose

The SQS collector can be configured to write any log to any table. Devo recommends use of a pre-built service that fits your logs. If the pre-built services do not fit, you should engage Devo professional services to create a custom service.

If you need to modify or filter logs, Devo recommends AWS Lambda.

Authorize It

Authorize SQS Data Access.
Add data to the S3 bucket. Preferably, the data should be in a consistent format. For example:
1. If the data is JSON objects, the keys of the JSON objects should be the same. Some objects can omit some keys.
2. If the data is comma separated value format, the number of columns must always be the same.

Gather Information

Get a log sample from the S3 bucket.
Determine if the S3 contents are compressed.
Choose a destination tag. Contact us for assistance checking for an existing tag or use a my.app tag.

Run It

Simple Template

In the Cloud Collector App, create an SQS Collector instance using this parameters template, replacing the values enclosed in < >.

{
  "inputs": {
    "sqs_collector": {
      "id": "<FIVE_UNIQUE_DIGITS>",
      "services": {
        "custom_service": {<OPTIONS>,
          "routing_template": "<DESTINATION TAG>"
        }
      },
      "credentials": {
              "aws_cross_account_role": "arn:<PARTITION>:iam::<YOUR_AWS_ACCOUNT_NUMBER>:role/<YOUR_ROLE>",
              "aws_external_id": "<EXTERNAL_ID>"
      },
      "region": "<REGION>",
      "base_url": "https://sqs.<REGION>.amazonaws.com/<YOUR_AWS_ACCOUNT_NUMBER>/<QUEUE_NAME>"
    }
  }
}

Collector customization options:

Flexible Example

{
  "global_overrides": {
    "debug": false
  },
  "inputs": {
    "sqs_collector": {
      "id": "12351",
      "enabled": true,
      "credentials": {
        "aws_access_key_id": "",
        "aws_secret_access_key": "",
        "aws_base_account_role": "arn:aws:iam::476382791543:role/devo-xaccount-cc",
        "aws_cross_account_role": "",
        "aws_external_id": ""
      },
      "ack_messages": true,
      "direct_mode": false,
      "do_not_send": false,
      "compressed_events": false,
      "base_url": "https://us-west-1.queue.amazonaws.com/id/name-of-queue",
      "region": "us-west-1",
      "sqs_visibility_timeout": 240,
      "sqs_wait_timeout": 20,
      "sqs_max_messages": 1,
      "services": {
        "custom_service": {
          "file_field_definitions": {
            "log_type": [
              {
                "operator": "split",
                "on": "/",
                "element": 0
              },
              {
                "operator": "replace",
                "to_replace": "_",
                "replace_with": ""
              }
            ]
          },
          "filename_filter_rules": [
            [
              {
                "type": "match",
                "pattern": "CloudTrail-Digest"
              }
            ],
            [
              {
                "type": "match",
                "pattern": "ConfigWritabilityCheckFile"
              }
            ]
          ],
          "encoding": "gzip",
          "send_filtered_out_to_unknown": false,
          "file_format": {
            "type": "line_split_processor",
            "config": {
              "json": true
            }
          },
          "record_field_mapping": {
            "event_simpleName": {
              "keys": [
                "event_simpleName"
              ]
            }
          },
          "routing_template": "destination tag",
          "line_filter_rules": [
            [
              {
                "source": "record",
                "key": "event_simpleName",
                "type": "match",
                "value": "EndOfProcess"
              }
            ],
            [
              {
                "source": "record",
                "key": "event_simpleName",
                "type": "match",
                "value": "DeliverLocalFXToCloud"
              }
            ]
          ]
        }
      }
    }
  }
}