Authorize SQS Data Access

Purpose

To use Devo to secure your AWS data, you will authorize Devo to receive the information. After completing these steps, you will have an S3 Bucket, SQS, SQS Access Policy, IAM Policy, Role, and Trust Policy in your AWS account. Data will be available to Devo and your policies will follow AWS recommended best practices.

Permissions

These AWS account permissions are required. This list may be incomplete.

s3:CreateBucket
s3:PutObject
s3:PutBucketNotification
sqs:CreateQueue
sqs:AddPermission
sqs:GetQueueUrl
sqs:SetQueueAttributes
iam:CreatePolicy
iam:CreateRole
iam:AttachRolePolicy
iam:UpdateAssumeRolePolicy
logs:CreateLogDelivery
logs:DeleteLogDelivery

Authorize it

Create a list of this information.
1. AWS Partition. For example, aws.
2. AWS Region. For example, us-east-1.
3. Your AWS Account. For example, 012345678901.
4. S3 Principal Service endpoint. This is almost always s3.amazonaws.com but GovCloud FIPS is a known exception.
5. S3 bucket name. You can make one up, such as BucketCloudTrailDevo if you are authorizing access to CloudTrail data.
6. SQS name. You can make one up, such as SQSCloudTrailDevo.
7. IAM policy name. Make this the same as the SQS name.
8. Role name. Make this the same as the SQS name.
9. External ID. Get the External ID from the Cloud Collector App. Select Keychains.
  
  The external ID is your Devo domain name and default sender’s syslog event load balancer, separated by a .. For example, if your Devo domain name is child@parent and you use the us.elb.relay.logtrust.net event load balancer, then your external ID is child@parent.us.elb.relay.logtrust.net.
10. Collector name. You can make one up. If your Devo domain is child@parent, the collector name could be child-parent-sqs-CloudTrailDevo.
Create an S3 bucket using the default settings. Use a separate bucket for each log format, if possible.
Create a standard SQS queue named devo_deadletter_queue. Use the default settings.
Create a second standard SQS queue using your queue name. This will be the queue used to send data to Devo.
Select the maximum retention period.

Set the following SQS access policy, replacing the values enclosed in < >.

{
  "Version": "2012-10-17",
  "Id": "<ENSURE_THE_ID_IS_UNIQUE>",
  "Statement": [
    {
      "Sid": "AllowS3toSendMessages",
      "Effect": "Allow",
      "Principal": {
        "Service": "<S3_PRINCIPAL_SERVICE_ENDPOINT>"
      },
      "Action": [
        "SQS:SendMessage"
      ],
      "Resource": "arn:<PARTITION>:sqs:<REGION>:<YOUR_AWS_ACCOUNT_NUMBER>:<QUEUE_NAME>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:<PARTITION>:s3:*:*:<BUCKET_NAME>"
        },
        "StringEquals": {
          "aws:SourceAccount": "<YOUR_AWS_ACCOUNT_NUMBER>"
        }
      }
    }
  ]
}

An example resource is arn:aws:sqs:us-east-1:0123456789:devologs.

Enable the dead-letter queue named devo_deadletter_queue.
Keep the defaults for the other SQS settings.
Return to the S3 Bucket Properties.
Select “Create event notification.”
Enable “All object create events.“
Select the SQS queue.
Create an IAM Policy.

Add this JSON Policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes",
        "sqs:ChangeMessageVisibility",
        "sqs:ReceiveMessage"
      ],
      "Resource": [
        "arn:<PARTITION>:sqs:<REGION>:<YOUR_AWS_ACCOUNT_NUMBER>:<QUEUE_NAME>",
        "arn:<PARTITION>:s3:::<BUCKET_NAME>/*"
      ]
    }
  ]
}

Click Next.
Name the policy.
Create the policy.
Create a role.
Select Custom trust policy.

Set the trust policy. If your partition is aws:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::476382791543:role/devo-xaccount-cc"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<EXTERNAL_ID>"
        }
      }
    }
  ]
}

If your partition is aws-us-gov:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws-us-gov:iam::210253767148:role/devo-xaccount-cc"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<EXTERNAL_ID>"
        }
      }
    }
  ]
}

Add the permissions policy you already created to the role.
Name the role.

Once the authorization process is complete, you can create a Devo collector and add data to the S3 Bucket.

Validate the Authorization

If there is no data being delivered to the S3 bucket yet and the collector is running, you may validate authorization by uploading an empty file to the bucket. In devo.collectors.out the collector will log

Number of messages in the queue: 0 and Number of messages in flight 1

followed by

the file size is 0 Bytes and has 1 to process

indicating correct authorization.