...
To run this collector, there are some configurations detailed below that you need to consider:
Configuration | Details |
---|---|
Credentials | There are several available options to define credentials. |
Service events | All the service events that are generated on AWS are managed by Cloudwatch. However, Devo’s AWS collector offers two different services that collect Cloudwatch events:
|
Audit events | For the S3+SQS approach (setting types as audits_s3) some previous configuration is required. |
Logs | Logs can be collected from different services. Depending on the type, some previous setups must be applied on AWS. |
Info |
---|
More information Refer to the Vendor setup section to know more about these configurations. |
...
You can use the AWS collector to retrieve data from the AWS APIs and send it to your Devo domain. Once the gathered information arrives at Devo, it will be processed and included in different tables in the associated Devo domain so users can analyze it.
Devo collector features
Feature | Details |
---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Data sources
Data Source | Description | API Endpoint | Collector service name | Devo table | Available from release |
---|---|---|---|---|---|
Service events | The different available services in AWS usually generate some information related to their internal behaviors, such as "a virtual machine has been started", "a new file has been created in an S3 bucket" or "an AWS lambda function has been invoked" and this kind of event can be triggered by no human interaction. The service events are managed by the The findings detected by |
| Generic events:
Security Hub events:
| Generic events:
Security Hub events:
|
|
Audit events | This kind of event is more specific because they are triggered by a human interaction no matter the different ways used: API, web interaction, or even the CLI console. The audit events are managed by the There are two ways to read Audit events:
| Via API:
Via S3+SQS:
|
|
|
|
Metrics | According to the standard definition, this kind of information is usually generated at the same moment is requested because it is usually a query about the status of a service (all things inside AWS are considered services). AWS makes something slightly different because what is doing is to generate metrics information every N time slots, such as 1 min, 5 min, 30 min, 1h, etc., even if no one makes a request (also is possible to have information every X seconds but this would require extra costs). The metrics are managed by the |
ListMetrics - Amazon CloudWatch After listing the metrics, GetMetricData - Amazon CloudWatch GetMetricStatistics - Amazon CloudWatch
|
|
|
|
Logs | Logs could be defined as information with a non-fixed structure that is sent to one of the available “logging” services, these services are There are some very customizable services, such as There are also some other services that can generate logs with a fixed structure, such as |
| Logs can be:
|
|
|
Vendor setup
...
There are some minimal requirements to set up this collector.
AWS console access: Credentials are required to access the AWS console.
Owner or Administrator permissions within the AWS console, or the fill access to configure AWS services.
Some manual actions are necessary in order to get all the required information or services and allow Devo to gather information from AWS. The following sections describe how to get the required AWS credentials and how to proceed with the different required setups depending on the gathered information type.
Credentials
It’s recommended to have available or create the following IAM policies before the creation of the IAM user that will be used for the AWS collector.
...
title | Policy details |
---|
Some collector services require the creation of some IAM policies before creating the IAM user that will be used for the AWS collector. The following table contains the details about the policies that could be used by the AWS collector:
...
Source type
...
AWS Data Bus
...
Recommended policy name
...
Variant
...
Details
...
Service events
...
CloudWatch Events
...
devo-cloudwatch-events
...
All resources
...
Tip |
---|
It’s not required the creation of any new policy due to there are not needed any permissions |
...
Audit events
Cisco Umbrella [Non-AWS service] | Cisco Umbrella is a cloud-driven Secure Internet Gateway (SIG) that leverages insights gained through the analysis of various logs, including DNS logs, IP logs, and Proxy logs, to provide a first line of defense. DNS logs record all DNS queries that are made through the Cisco Umbrella DNS resolvers. These logs contain data about the DNS queries originating from your network, requested domain names and the IP address of the requester. IP logs capture all IP-based communications that occur through the network. These logs store details such as the source and destination IP addresses, ports and protocols used. Proxy logs are generated when users access web resources through the Cisco Umbrella intelligent proxy. They contain detailed information on the web traffic including the URL accessed, the method of access (GET, POST, etc.), the response status, etc | Via S3+SQS:
|
|
|
|
Vendor setup
Anchor | ||||
---|---|---|---|---|
|
There are some minimal requirements to set up this collector.
AWS console access: Credentials are required to access the AWS console.
Owner or Administrator permissions within the AWS console, or the fill access to configure AWS services.
Some manual actions are necessary in order to get all the required information or services and allow Devo to gather information from AWS. The following sections describe how to get the required AWS credentials and how to proceed with the different required setups depending on the gathered information type.
Credentials
It’s recommended to have available or create the following IAM policies before the creation of the IAM user that will be used for the AWS collector.
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Some collector services require the creation of some IAM policies before creating the IAM user that will be used for the AWS collector. The following table contains the details about the policies that could be used by the AWS collector:
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Depending on which source types are collected, one or more of the policies described above will be used. Once the required policies are created, each one must be associated with an IAM user. To create it, visit the AWS Console and log in with a user account with enough permissions to create and access AWS structures: Go to IAM → Users Click Add users button. Enter the required value in the filed User name. Enable the checkbox Access key - Programmatic access. Click on Next: Tags button.
|
Expand | ||
---|---|---|
| ||
Depending on which source types are collected, one or more of the policies described above will be used. Once the required policies are created, each one must be associated with an IAM user. To create it, visit the AWS Console and log in with a user account with enough permissions to create and access AWS structures:
|
...
Expand | ||||
---|---|---|---|---|
| ||||
In case you don't want to share your credentials with Devo, you should add some parameters to the configuration file. In the credentials section, instead. of sharing
|
Service Events
Cloudwatch manages all the service events that have been generated on AWS. However, Devo’s AWS Collector offers two different services that collect Cloudwatch Events:
sqs-cloudwatch-consumer
: This service is used to collect Security Hub events.service-events-all
: This service is used to collect events from the rest of the services on AWS
Info |
---|
Service events Some previous configurations are required if you want to use any of these services. The AWS services generate service events per region, so the following instructions should be applied in each region where the collecting information is required. There are some structures that you need to create for collecting these service events: |
If you want to create them manually, click on each one to follow the steps.
Expand | ||
---|---|---|
title | SQS FIFO queue creation
Service Events
Cloudwatch manages all the service events that have been generated on AWS. However, Devo’s AWS Collector offers two different services that collect Cloudwatch Events:
sqs-cloudwatch-consumer
: This service is used to collect Security Hub events.service-events-all
: This service is used to collect events from the rest of the services on AWS
Info |
---|
Service events Some previous configurations are required if you want to use any of these services. The AWS services generate service events per region, so the following instructions should be applied in each region where the collecting information is required. There are some structures that you need to create for collecting these service events: |
If you want to create them manually, click on each one to follow the steps.
Expand | ||
---|---|---|
| ||
|
Expand | ||
---|---|---|
| ||
|
Steps to enable Audit Events
No actions are required in Cloudtrail Service for retrieving this kind of information when the API approach is used (setting types as audit_apis).
For the S3+SQS approach (setting types as audits_s3) some previous configuration is required. Find a complete description of how to create an S2 +SQS pipeline here.
Steps to enable Metrics
No actions are required in CloudWatch Metrics service for retrieving this kind of information.
Steps to enable Logs
Logs can be collected from different services. Depending on the type, some previous setups must be applied on AWS:
Expand | ||
---|---|---|
| ||
No actions are required in this service for retrieving this kind of information. |
Expand | ||
---|---|---|
| ||
Expand | ||
title | VPC FlowBefore enabling the generation of these logs some structures must be created: one bucket in the S3 service and one FIFO queue in the SQS service. Follow the steps to create those structures manually: Create SQS Stadard queue
Create or configure S3 bucket
|
Expand | ||
---|---|---|
| ||
|
Steps to enable Audit Events
No actions are required in Cloudtrail Service for retrieving this kind of information when the API approach is used (setting types as audit_apis).
For the S3+SQS approach (setting types as audits_s3) some previous configuration is required. Find a complete description of how to create an S2 +SQS pipeline here.
Steps to enable Metrics
No actions are required in CloudWatch Metrics service for retrieving this kind of information.
Steps to enable Logs
Logs can be collected from different services. Depending on the type, some previous setups must be applied on AWS:
Expand | ||
---|---|---|
| ||
No actions are required in this service for retrieving this kind of information. |
Create Flow Log
|
Expand | ||
---|---|---|
| ||
Before enabling the generation of these logs some structures must be created: one bucket in the S3 service and one FIFO queue in the SQS service. Follow the steps to create those structures manuallyFor the manual creation of these required structures, please follow the next steps: Create SQSStadardStandard queue
Create or configure S3 bucket
Allow Loggin in CloudfrontBefore enabling the generation of these logs some structures must be created: one bucket in the S3 service and one FIFO queue in the SQS service. For the manual creation of these required structures, please follow the next steps: Create SQS Standard queue
| ||
Expand | ||
| ||
|
Steps to enable Cisco Umbrella logs
Action | Steps |
---|---|
|
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
Allow Loggin in Cloudfront
Go to Cloudfront service.
Click on ID field link of the target Distribution item (for accessing to Distributing Settings options).
Click on the Edit button.
In Logging choose the value On.
In the Bucket for Logs field value set the ARN of the previously created S3 bucket (Saved in a previous step).
Click on Yes and then click on the Edit button.
Enable |
|
Minimum configuration required for basic pulling
...
Info |
---|
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details. |
Setting | Details |
---|---|
| This is the account identifier for AWS. More info can be found in the section Using a user account and local policies. |
| This is the secret (kind of a password) for AWS. More info can be found in the section Using a user account and local policies. |
| This allows assuming a role with limited privileges to access AWS services. More info can be found in the sections Assuming a role (self-account) and/or Assuming a role (cross-account). |
| This allows assuming a role on another AWS account with limited privileges to access AWS services. More info can be found in the section Assuming a role (cross-account). |
| This is an optional field that provides additional security to the assuming role operation on cross-accounts. More info can be found in the section Assuming a role (cross-account). |
Info |
---|
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method. |
...
Depending on how did you obtain your credentials, you will have to either fill in or delete the following properties on the JSON credentials
configuration block.
Authentication Method | access_key | access_secret | base_assume_role | target_assume_role | assume_role_external_id | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Access Key / Access Secret |
|
|
|
|
| ||||||||||||||||||
Assume role (self-account) |
|
|
|
|
| ||||||||||||||||||
Assume role (cross-account) |
|
|
|
|
|
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Rw ui tabs macro | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector:
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in
Editing the config.yaml file
Replace the placeholders with your required values following the description table below:
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Use the following command to add the Docker image to the system:
The Docker image can be deployed on the following services: DockerExecute the following command on the root directory
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the
To run the container using docker-compose, execute the following command from the
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration. |
...
Expand | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components:
Setup outputA successful run has the following output messages for the setup module:
Puller outputA successful initial run has the following output messages for the puller module:
|
Cisco Umbrella (via S3+SQS)
This service reads logs from a Cisco Umbrella managed bucket via the S3+SQS pipeline. Cisco provides a way to deposit logging data into a S3 bucket
Devo categorization and destination
There are three types of events: dnslogs, iplogs and proxylogs. Cisco stores them in different paths depending on the event type. The collector will ingest them towards the following tables:
dnslogs:
sig.cisco.umbrella.dns
iplogs:
sig.cisco.umbrella.ip
proxylogs:
sig.cisco.umbrella.proxy
In case Cisco starts sending other type of events to s3, they will go to:
sig.cisco.umbrella.unknown
Collector operations
This section is intended to explain how to proceed with the specific operations of this collector.
...
Expand | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Common for all the services using the S3+SQS pipeline
Audit (via API)
Custom Logs
Metrics
|
Change log for v1.x.x
...
Release
...
Released on
...
Release type
...
Details
...
|
Change log
Release | Released on | Release type | Details | Recommendations | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| New features:
|
| |||||||||||||||||||
|
| Improvements
|
| |||||||||||||||||||
|
| Bug Fixes:
|
| |||||||||||||||||||
|
| New features:
Improvements:
Bug Fixes:
|
|
v1.4.1
Status | ||||
---|---|---|---|---|
|
Bug Fixes:
Fixed a bug that prevented the use of the Assumed Role authentication method.
Fixed a bug that prevented session renewal when using any of the Assume Authentication methods:
Assume Role
Cross Account
Recommended version