| Table of Contents | ||||
|---|---|---|---|---|
|
...
Amazon Web Services (AWS) provides on-demand cloud computing platforms and APIs to individuals and companies. Each available AWS service generates information related to different aspects of its functionality. The available data types include service events, audit events, metrics, and logs.
You can use the AWS collector to retrieve data from the AWS APIs and send it to your Devo domain. Once the gathered information arrives at Devo, it will be processed and included in different tables in the associated Devo domain so users can analyze it.
Data source description
From the monitoring point of view, AWS generates the following types of information:
...
Devo collector service name | Complete service name | CloudWatch filter used | CloudTrail source filter used | Metrics namespace used | Description | Service events | Audit events | Metrics | Logs |
|---|---|---|---|---|---|---|---|---|---|
| All service events |
| N/A | N/A | This service will collect all service events information available in the CloudWatch service, no matter the source defined in the event. | ✓ | X | X | X |
| All audit events | N/A |
| N/A | This service will collect all audit events information available in the CloudTrail service, no matter the source defined in the event. | X | ✓ | X | X |
| All metrics | N/A | N/A |
| This service will collect all metric information from CloudWatch service. Metrics from all the available metric namespaces will be retrieved. | X | X | ✓ | X |
| CloudWatch Logs | N/A | N/A | N/A | This service will collect the different “Log Streams” that are part of a “Log Group” from the CloudWatch Logs service. Since it is common to have more than one “Log Group” defined, this will require creating one | X | X | X | ✓ |
| Non-CloudWatch Logs | N/A | N/A | N/A | This service will collect data from the following services VPC Flow Logs and CloudFront Logs. | X | X | X | ✓ |
| Service events generated by CloudWatch Events service | Check more info here. | N/A | N/A | This service will collect all Security Hub findings that have been sent to CloudWatch, no matter the source defined in the finding. | ✓ | X | X | X |
| Info |
|---|
In the |
...
Once the data source is configured, you can send us the required information and we will host and manage the collector for you (Cloud collector), or you can host the collector in your own machine using a Docker image (On-premise collector).
| Rw ui tabs macro |
|---|
| Rw tab | ||
|---|---|---|
|
| Rw tab | ||
|---|---|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the AWScollector:
| Code Block |
|---|
<any_directory>
└── devo-collectors/
└── aws/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
└── config/
└── config-aws.yaml |
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/aws/certs. Learn more about security credentials in Devo here.
Editing the config-aws.yaml file
In the config-aws.yaml file, replace the <short_unique_identifier>, <access_key_value>, <access_secret_value>, <region_*> and <log_group_value_*> values and enter the ones that you got in the previous steps. In the <short_unique_identifier>, <region_*> and <log_group_value_*> placeholders, enter the value that you choose.
| Code Block |
|---|
globals:
debug: false
id: not_used
name: aws
persistence:
type: filesystem
config:
directory_name: state
outputs:
devo_1:
type: devo_platform
config:
address: collector-eu.devo.io
port: 443
type: SSL
chain: chain.crt
cert: <your_domain>.crt
key: <your_domain>.key
inputs:
aws:
id: <short_unique_identifier>
enabled: true
requests_per_second: 5
autoconfig:
enabled: true
refresh_interval_in_seconds: 600
credentials:
access_key: <access_key_value>
access_secret: <access_secret_value>
services:
service-events-all:
#tag: my.app.aws_service_events
#sqs_queue_name: <custom_queue_name>.fifo
regions:
- <region_a>
- <region_b>
- <region_c>
audit-events-all:
#tag: my.app.aws_audit_events
regions:
- <region_a>
- <region_b>
- <region_c>
metrics-all:
#tag: my.app.aws_metrics
regions:
- <region_a>
- <region_b>
- <region_c>
non-cloudwatch-logs:
#tag: my.app.aws_non_cwl
#cloudfrontlogs_sqs_queue_name: <custom_queue_name>
#vpcflowlogs_sqs_queue_name: <custom_queue_name>
regions:
- <region_a>
- <region_b>
- <region_c>
cwl_1:
#tag: my.app.aws_cwl
types:
- logs
log_group: <log_group_a>
regions:
- <region_a>
- <region_b>
- <region_c>
cwl_2:
#tag: my.app.aws_cwl
types:
- logs
log_group: <log_group_b>
regions:
- <region_a>
- <region_b>
- <region_c> |
| Info |
|---|
By default, there are several SQS queue name patterns that the collector will try to use for retrieving information from AWS. Depending on the info type, the following patterns exist:
|
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Collector Docker image | SHA-256 hash |
|---|---|
|
Use the following command to add the Docker image to the system:
| Code Block |
|---|
gunzip -c collector-aws-docker-image-<version>.tgz | docker load |
| Info |
|---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace " |
The Docker image can be deployed on the following services:
Docker
Docker Compose
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/aws/
| Code Block |
|---|
docker run \ --name collector-aws \ --volume $PWD/certs:/devo-collector/certs \ --volume $PWD/config:/devo-collector/config \ --volume $PWD/state:/devo-collector/state \ --env CONFIG_FILE=config-aws.yaml \ --rm -it docker.devo.internal/collector/aws:<version> |
| Note |
|---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/aws/ directory.
| Code Block |
|---|
version: '3'
services:
collector-aws:
build:
context: .
dockerfile: Dockerfile
image: docker.devo.internal/collector/aws:${IMAGE_VERSION:-latest}
container_name: collector-aws
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config-aws.yaml} |
| Rw tab | ||
|---|---|---|
|
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.