AWS collector

Information that happens at a specific timestamp. This information will always be linked to that timestamp, and can be categorized into two different subtypes:

Service events

The different available services usually generate information related to their internal behaviors, such as A virtual machine has been started, A new file has been created in an S3 bucket or An AWS lambda function has been invoked.

Note that this type of event can be triggered with no human interaction. These kinds of events are managed by the CloudWatch Events service (CWE). Recently, AWS has created a new service called Amazon EventBridge that will replace the CWE service.
The findings detected by AWS Security Hub are also managed by CloudWatch Events (CWE).

Audit events

These events are more specific because they need human interaction no matter the way used to retrieve them (API, web interaction, or even CLI command).

These events are managed by the CloudTrail service.

According to the standard definition, this kind of information is usually generated at the exact moment it is requested because it is typically a query related to the status of a service (everything inside AWS is considered a service).

AWS makes something slightly different because it generates metrics information every N time slots, such as 1 min, 5 min, 30 min, 1h, etc., even if no one makes a request.

This kind of information is managed by the CloudWatch Metrics service (CWM).

Logs can be defined as information with a non-fixed structure that is sent to one of the available logging services. These services are CloudWatch Logs and S3.

There are some very customizable services, such as AWS Lambda, or even any developed application which is deployed inside an AWS virtual machine (EC2), that can generate custom log information. This kind of information is managed by the CloudWatch Logs service (CWL) and also by the S3 service.

There are also some other services that can generate logs with a fixed structure, such as VPC Flow Logs or CloudFront Logs. These kinds of services require one special way of collecting their data.

Depending on which source types are collected, one or more of the policies described above will be used.

Once the required policies are created, each one must be associated with an IAM user. To create it, visit the AWS Console and log in with a user account with enough permissions to create and access AWS structures:

1. Go to IAM → Users.

2. Click the Add users button.

3. Enter the required value in the field User name.

4. Enable the checkbox Access key - Programmatic access.

5. Click the Next: Permissions button.

6. Choose the box with the text Attach existing policies directly.

7. Use the search box to locate all your required policies and check the boxes at the left of each policy name.

8. Click Next: Tags. Optionally, add any desired tags to the new user.

9. Click Next: Review.

10. Click the Create user button.

11. A new Access Key ID and Secret Access Key will be created. You can click the Download .csv button to download a copy to your local or you can copy the values shown on the screen. These will be used as the AWS collector credentials.

12. Finally, click the Close button.

It is a best practice to assume roles that are granted just the required privileges to perform an action. If the customer does not want to use their own AWS user to perform these actions required by the collector (because it has far more privileges than required), they can use this option. Note that this option requires the use of AWS account credentials. To avoid sharing those credentials, check the Cross Account option below.

The customer must attach the required policies in AWS to the role that is going to be assumed. For more information about the AssumeRole feature, check the AWS documentation.

Regarding configuration, these are the fields required to use this way of authentication:

...,
"credentials":{
  "access_key": "<CUSTOMER_AWS_ACCOUNT_ACCESS_KEY>",
  "access_secret": "<CUSTOMER_AWS_ACCOUNT_SECRET_ACCESS_KEY>",
  "base_assume_role": "arn:aws:iam::<CUSTOMER_AWS_ACCOUNT_ID>:role/<ROLE_TO_BE_ASSUMED>"
}
...,

• access_key is the Access Key ID provided by AWS during the user creation process to those users that ask for programmatic access.

• access_secret is the Secret Access Key provided by AWS during the user creation process to those users that ask for programmatic access. This value is shown once, so it must be saved upon creation.

• base_assume_role is the ARN of the role that is going to be assumed by the user authenticated with the parameters above, access_key and access_secret. This role has to be properly granted to allow the actions the collector is going to perform.

If the customer does not want to share their credentials with Devo, there is another way to run the collector. It is called Cross Account and AssumeRole functionality should be used in this case and it is explained step by step in this article.

Besides, some parameters must be added to the configuration file (config.json or equivalent config.yaml for on-prem). In the credentials section, instead of sharing access_key and access_secret, these other parameters must be used:

...,
"credentials":{
  "base_assume_role": "arn:aws:iam::<BASE_SYSTEM_AWS_ACCOUNT_ID>:role/<BASE_SYSTEM_ROLE>",
  "target_assume_role": "arn:aws:iam::<CUSTOMER_AWS_ACCOUNT_ID>:role/<CUSTOMER_ROLE_TO_BE_ASSUMED>",
  "assume_role_external_id": "<OPTIONAL__ANY_STRING_YOU_WANT>"
}
...,

• base_assume_role is the ARN of the role that is going to be assumed by the profile bound to the machine/instance where the collector is running. As explained in the link above, this role is going to be trusted by the customer’s AWS account, so it can assume the role in the target account. That role assumed from the customer’s account will allow the collection of data without the need of sharing the credentials. This role already exists in Devo’s AWS account and to deploy the collector on Devo’s Collector Server its value must be arn:aws:iam::837131528613:role/devo-xaccount-cs-role.

• target_assume_role is the ARN of the role in the customer’s AWS account. That role will allow the collector to have access to the resources specified in that role. To keep your data secure, please, use policies that grant just the necessary permissions.

• assume_role_external_id is an optional parameter to add more security to this Cross Account operation. This value should be a string added in the request to assume the customer’s role.

1. Go to Simple Queue Service and click Create queue.

2. In the Details section, choose the FIFO queue type and set the Name field value you prefer (it must finish with the .fifo suffix).

3. In the Configuration section, set the Message retention period field value to 5 Days. Be sure that the Content-based deduplication checkbox is marked and leave the rest of the options with their default values.

4. In the Access policy section, choose the method Basic and select the option Only the queue owner to receive and send permissions.

5. Optionally, in the Tags section, you can create a tag with Key usedBy and Value devo-collector.

6. Click Create queue.

1. Go to CloudWatch → Rules and click Create rule.

2. In the Event Source section, select Event Pattern and Build event pattern to match events by service.

3. In the Service Name field, enter the service to be monitored (Check the note below for Security Hub Findings)

4. In the Event Type field, choose All Events (Check the note below for Security Hub Findings)

5. In the Targets section, click Add target and choose SQS queue as the target type.

6. In the Queue dropdown, choose the previously created queue.

7. In the Message group ID field, set the value devo-collector.

8. Then, click Configure details.

9. In the Rule definition section, set the Name you prefer. Be sure that State checkbox is marked.

1. Go to Simple Queue Service and click Create queue.

2. In the Details section, choose the Standard queue type and set the Name field value you prefer.

3. In the Configuration section, set the Message retention period field value to 5 Days and leave the rest of the options with their default values.

4. In the Access policy section, choose the method Advanced and replace "Principal": {"AWS":"<account_id>"} by "Principal": "*" Leave the rest of the JSON as default.

5. Optionally, in the Tags section, you can create a tag with Key usedBy and Value devo-collector.

6. Click Create queue.

1. Go to S3 and click Create bucket.

2. Set the preferred value in the Bucket name field.

3. Choose the required Region value and click Next.

4. Optionally, in the Tags section, you can create a tag with Key usedBy and Value devo-collector. Leave the rest of the fields with their default values and click Next.

5. Click Create bucket.

6. Mark the checkbox next to the previously created S3 bucket.

7. In the popup box, click Copy Bucket ARN and save the content. You will need it later.

8. In the S3 bucket list, click the previously created bucket name link.

9. Click the Properties tab, then click the Events box.

10. Click Add notification.

11. Set the preferred value in the Name field.

12. Select the All object create events checkbox.

13. In the Send to field, select SQS Queue.

14. Select the previously created SQS queue in the SQS field.

Once the required AWS structures are created, go to the VPC service and follow these steps:

1. Select any available VPC (or create a new one).

2. Go to the Flow Logs tab and click Create flow log.

3. Choose the preferred Filter value and the required Maximum aggregation interval value.

4. In the Destination field, select Send to an S3 bucket.

5. In the S3 bucket ARN field set the ARN of the previously created S3 bucket.

6. Make sure that the Format field has the value AWS default format.

7. Optionally, in the Tags section, you can create a tag with Key usedBy and Value devo-collector.

8. Finally, click Create.

1. Go to Simple Queue Service and click Create queue.

2. In the Details section, choose the Standard queue type and set the Name field value you prefer.

3. In the Configuration section, set the Message retention period field value to 5 Days and leave the rest of the options with their default values.

4. In the Access policy section, choose the method Advanced and replace "Principal": {"AWS":"<account_id>"} by "Principal": "*" Leave the rest of the JSON as default.

5. Optionally, in the Tags section, you can create a tag with Key usedBy and Value devo-collector.

6. Click Create queue.

1. Go to S3 and click Create bucket.

2. Set the preferred value in the Bucket name field.

3. Choose the required Region value and click Next.

4. Optionally, in the Tags section, you can create a tag with Key usedBy and Value devo-collector. Leave the rest of the fields with their default values and click Next.

5. Click Create bucket.

6. Mark the checkbox next to the previously created S3 bucket.

7. In the popup box, click Copy Bucket ARN and save the content. You will need it later.

8. In the S3 bucket list, click the previously created bucket name link.

9. Click the Properties tab, then click the Events box.

10. Click Add notification.

11. Set the preferred value in the Name field.

12. Select the All object create events checkbox.

13. In the Send to field, select SQS Queue.

14. Select the previously created SQS queue in the SQS field.

Once the required AWS structures are created, go to the CloudFront service and follow these steps:

1. Click the ID of the target Distribution item and access the Distributing Settings options. Then, click Edit.

2. In the Logging field, select On.

3. In the Bucket for Logs field, enter the ARN of the previously created S3 bucket.

4. Finally, click the Yes, Edit button.