Amazon Web Services (AWS) provides on-demand cloud computing platforms and APIs to individuals and companies. Each available AWS service generates information related to different aspects of its functionality. The available data types include service events, audit events, metrics, and logs.
You can use the AWS collector to retrieve data from the AWS APIs and send it to your Devo domain. Once the gathered information arrives at Devo, it will be processed and included in different tables in the associated Devo domain so users can analyze it.
From the monitoring point of view, AWS generates the following types of information:
Information that happens at a specific timestamp. This information will always be linked to that timestamp, and can be categorized into two different subtypes: Service eventsThe different available services usually generate information related to their internal behaviors, such as A virtual machine has been started, A new file has been created in an S3 bucket or An AWS lambda function has been invoked. Note that this type of event can be triggered with no human interaction. These kinds of events are managed by the CloudWatch Events service (CWE). Recently, AWS has created a new service called Amazon EventBridge that will replace the CWE service. Audit eventsThese events are more specific because they need human interaction no matter the way used to retrieve them (API, web interaction, or even CLI command). These events are managed by the CloudTrail service. |
According to the standard definition, this kind of information is usually generated at the exact moment it is requested because it is typically a query related to the status of a service (everything inside AWS is considered a service). AWS makes something slightly different because it generates metrics information every N time slots, such as 1 min, 5 min, 30 min, 1h, etc., even if no one makes a request. This kind of information is managed by the CloudWatch Metrics service (CWM). |
Logs can be defined as information with a non-fixed structure that is sent to one of the available logging services. These services are CloudWatch Logs and S3. There are some very customizable services, such as AWS Lambda, or even any developed application which is deployed inside an AWS virtual machine (EC2), that can generate custom log information. This kind of information is managed by the CloudWatch Logs service (CWL) and also by the S3 service. There are also some other services that can generate logs with a fixed structure, such as VPC Flow Logs or CloudFront Logs. These kinds of services require one special way of collecting their data.
|
Some manual actions are necessary in order to get all the required information/services and allow the Devo collector to gather the information from AWS.
The following sections describe how to get the required AWS credentials and how to proceed with the different required setups depending on the gathered information type.
Because there are several options about how to create the credentials, they will be detailed only in two different approaches.
There are several available options to define credentials, but we will only cover some of them.
It’s recommended to have available or create the following IAM policies before the creation of the IAM user that will be used for the AWS collector.
|
Depending on which source types are collected, one or more of the policies described above will be used. Once the required policies are created, each one must be associated with an IAM user. To create it, visit the AWS Console and log in with a user account with enough permissions to create and access AWS structures:
|
All the service events that are generated on AWS are managed by Cloudwatch. However, Devo’s AWS collector offers two different services that collect Cloudwatch events:
sqs-cloudwatch-consumer
- This service is used to collect Security Hub events.
service-events-all
-This service is used to collect events from the rest of the services on AWS.
The AWS services generate service events per region, so the next instructions should be applied in each region where the collecting of information is required (use the same values for all your configured regions). |
In order to collect these service events, there are some structures that must be created: one FIFO queue in the SQS service and one Rule+Target in the CloudWatch service.
If the auto-setup functionality is enabled in the configuration and the related credentials have enough permissions to create all the required AWS structures, the following steps are not required. |
For a manual creation of these required structures, follow the next steps (click to expand):
|
|
No actions are required in the Cloudtrail service to retrieve this kind of information.
No actions are required in the CloudWatch Metrics service to retrieve this kind of information.
Logs can be collected from different services. Depending on the service type, you may need to apply some settings on AWS:
No actions are required in this service to retrieve this kind of information.
Before enabling the generation of these logs, you must create one bucket in the S3 service and one FIFO queue in the SQS service. For a manual creation of these required structures, follow these steps (click to expand):
|
|
Once the required AWS structures are created, go to the VPC service and follow these steps:
|
Before enabling the generation of these logs, you must create one bucket in the S3 service and one FIFO queue in the SQS service. For a manual creation of these required structures, follow these steps (click to expand):
|
|
Once the required AWS structures are created, go to the CloudFront service and follow these steps:
|
The following tables show details about the predefined services available to be used in the collector configuration.
Devo collector service name | Complete service name | CloudWatch filter used | CloudTrail source filter used | Metrics namespace used | Description | Service events | Audit events | Metrics | Logs |
---|---|---|---|---|---|---|---|---|---|
| All service events |
| N/A | N/A | This service will collect all service events information available in the CloudWatch service, no matter the source defined in the event. | ✓ | X | X | X |
| All audit events | N/A |
| N/A | This service will collect all audit events information available in the CloudTrail service, no matter the source defined in the event. | X | ✓ | X | X |
| All metrics | N/A | N/A |
| This service will collect all metric information from CloudWatch service. Metrics from all the available metric namespaces will be retrieved. | X | X | ✓ | X |
| CloudWatch Logs | N/A | N/A | N/A | This service will collect the different “Log Streams” that are part of a “Log Group” from the CloudWatch Logs service. Since it is common to have more than one “Log Group” defined, this will require creating one | X | X | X | ✓ |
| Non-CloudWatch Logs | N/A | N/A | N/A | This service will collect data from the following services VPC Flow Logs and CloudFront Logs. | X | X | X | ✓ |
| Service events generated by CloudWatch Events service | Check more info here. | N/A | N/A | This service will collect all Security Hub findings that have been sent to CloudWatch, no matter the source defined in the finding. | ✓ | X | X | X |
In the |
The values entered in |
Depending on the data type chosen for collecting, the following service definitions could be added to the configuration inside the services section. The following are common properties that all services have:
regions
(mandatory) - It must be a list with valid target region names to be used when collecting data. One processing thread will be created per region. See more info about the available regions here.
request_period_in_seconds
(optional) - The period in seconds to be used between pulling executions (default value: 60)
pull_retries
(optional) - Number of retries that will be executed when a pulling error occurs (default value: 3)
tag
(optional) - Used for sending the data to a table different from the default one (in the configuration examples, they appear as commented lines).
These service definitions can be used for collecting in a global way the different data types available in AWS.
This is the configuration to be used when any service event needs to be collected from AWS, except Security Hub.
service-events-all: #tag: my.app.aws_service_events cloudwatch_sqs_queue_name: <queue_name> #auto_event_type: <bool> regions: - <region_a> - <region_b> - <region_c> |
The default target table is cloud.aws.cloudwatch.events |
This is the configuration to be used when Security Hub events need to be collected.
sqs-cloudwatch-consumer: #tag: <str> cloudwatch_sqs_queue_name: <queue_name> #auto_event_type: <bool> regions: - <region_a> - <region_b> - <region_c> |
The SQS queue name is required |
The default target table is cloud.aws.securityhub.findings |
There are two ways to get audit events. In case just a few events are going to be generated in the platform, using the API may be enough. However, when mid or high volumes are expected, saving those audit events in an S3 bucket would be the best choice. In this case, an SQS queue should be created to consume those events from the collector.
This is how the config file should be defined to retrieve audit events via API:
audit-events-all: #tag: <str with {placeholders}> #types: #- audits_api <str> #auto_event_type: <bool> #request_period_in_seconds: <int> #start_time: <datetime_iso8601_format> #drop_event_names: ["event1", "event2"] <list of str> regions: - <region_a> - <region_b> - <region_c> |
Field | Type | Mandatory | Description |
---|---|---|---|
| string | no | Tag or tag format to be used. i.e.:
|
| list of strings (in yaml format) | no | Enable/Disable modules only when several modules per service are defined. To get audit events from API, this field should be set to |
| integer | no | Period in seconds used between each data pulling, this value will overwrite the default value (60 seconds) |
| datetime | no | Datetime from which to start collecting data. It must match ISO-8601 format. |
| boolean | no | Used to enable the auto categorization of message tagging. |
| list of strings | no | If the value in i.e. if this parameter is populated with the next values |
| list of strings (in yaml format) | yes, if defined in the “Collector definitions”. | Property name ( |
On the other hand, if S3 + SQS is the chosen option to get the audit events, the config file should match the following format:
audit-events-all: #tag: <str with {placeholders}> #types: #- audits_s3 <str> #request_period_in_seconds: <int> #start_time: <datetime_iso8601_format> #auto_event_type: <bool> audit_sqs_queue_name: <str> #s3_file_type_filter: <str (RegEx)> #use_region_and_account_id_from_event: <bool> regions: - region_a <str> - region_b <str> - region_c <str> |
The default target table is cloud.aws.cloudwatch.events |
Field | Type | Mandatory | Description |
---|---|---|---|
| string | no | Tag or tag format to be used. i.e.:
|
| list of strings (in yaml format) | no | Enable/Disable modules only when several modules per service are defined |
| integer | no | Period in seconds used between each data pulling, this value will overwrite the default value (60 seconds) |
| datetime | no | Datetime from which to start collecting data. It must match ISO-8601 format. |
| boolean | no | Used to enable the auto categorization of message tagging. |
| string | yes | Name of the SQS queue to read from. |
| string | no | RegEx to retrieve proper file type from S3 |
| bool | no | If |
| list of strings (in yaml format) | yes, if defined in the “Collector definitions”. | Property name ( |
metrics-all: #tag: my.app.aws_metrics regions: - <region_a> - <region_b> - <region_c> |
The default target table is cloud.aws.cloudwatch.metrics |
An entry per Log Stream that wanted to be processed must be defined. In this example, two different entries have been created (cwl_1
, cwl_2
) for processing the Log Streams called /aws/log_stream_a
and /aws/log_stream_b
cwl_1: #tag: my.app.aws_cwl types: -logs log_group: /aws/log_stream_a regions: - <region_a> - <region_b> - <region_c> cwl_2: #tag: my.app.aws_cwl types: -logs log_group: /aws/log_stream_b regions: - <region_a> - <region_b> - <region_c> |
As shown in the examples, the types list must be fixed with the log values. |
The default target table is cloud.aws.cloudwatch.logs |
non-cloudwatch-logs: #tag: my.app.aws_cwl #vpcflowlogs_sqs_queue_name: <custom_queue_a> #cloudfront_sqs_queue_name: <custom_queue_b> #auto_event_type: <bool> regions: - <region_a> - <region_b> - <region_c> |
The default target tables are cloud.aws.vpc.flowlogs and cloud.aws.cloudfront. |
The default existing expected SQS queue names for this service are |
The properties |
Once the data source is configured, you can send us the required information and we will host and manage the collector for you (Cloud collector), or you can host the collector in your own machine using a Docker image (On-premise collector).
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration. This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the AWS collector:
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in ![]() Editing the config-aws.yaml fileIn the config-aws.yaml file, replace the
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Use the following command to add the Docker image to the system:
The Docker image can be deployed on the following services:
DockerExecute the following command on the root directory
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the
|