AWS S3 + SQS Devo collector setup (CloudTrail example)

Create S3 bucket and set up data feed

The following will be set up in this section:

• S3 logging bucket

• CloudTrail data being sent into S3 bucket

Create S3 bucket

Setup CloudTrail to log in to S3 bucket

Create SQS queue and enable S3 event notifications

SQS provides the following benefits:

• Built-in retrying on failure of processing a message

• Dead letter queueing (if enabled when setting up SQS queue)

• Allows for downstream outage without loss of state of processing

• Allows for parallelisation of workers in event of very high volume data

• Guaranteed at least once delivery (S3 and SQS guarantees)

• Ability to have multiple S3 buckets send events to the same SQS queue and even those in other accounts via S3 event notifications to SNS to SQS in target account

(Optional) Using event notifications with SNS

Sending S3 event notifications to SNS may be beneficial/required to some teams if they are using the event notifications in the buckets in multiple applications. This is fully supported as long as the original S3 event notification message gets passed through SNS transparently to SQS. You will not need to follow the steps to set up event notifications to SQS, but you may follow the Amazon documentation here to set up the following:

• S3 Event Notifications → SNS

• SNS → SQS

A brief write up of this architecture can be found here.

Create the SQS queue for specified technology type

In this example, we will continue by setting up a SQS queue for our CloudTrail technology logs.

Setup S3 event notifications

Enabling SQS and S3 access using a cross-account IAM role

In order to allow the Devo collector server to pull in data from your AWS environment, you will need an IAM cross account role in your account. You will have to provide this role ARN to Devo.

Create an IAM Policy

This IAM policy will do the following:

• Allow the role to read messages off the SQS queue and acknowledge (delete) them off the queue after successfully processing the message.

• Retrieve the S3 object referenced in the SQS message so that Devo can read and process the message into the system.

• Limited to only the resources specified (minimal permissions).

Cross account role

Cross account roles let roles/users from another AWS account (in this case, the Devo collector server AWS account) access to assume a role in your account. This sidesteps the need to exchange credentials, as they are still stored separately in their respective accounts and AWS authenticates the identities. For more information check this article.

Information provided to Devo

At the end of this process, you will need to provide the following tidbits of information to Devo in order to set up the integration:

• Technology type that you will be consuming or log format (how should we process and tag this data, in this example CloudTrail logs).

• SQS queue URL.

• Cross account role ARN (for example, arn:aws:iam::<YOUR-ACCOUNT-ID>:role/devo-xs-collector-role) or external ID if used for cross account role policy.

Once this information is provided and Devo confirms or finishes creating a processor for your technology type, we will deploy and start consuming data off of the queue and from S3.