Versions Compared

Version Old Version 43 New Version Current
Changes made by

Juan Tomás Alonso Nieto

Laszlo Frazer

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents
maxLevel2
typeflat
Info

Devo recommends using the AWS SQS collector instead of the AWS collector. SQS improves ease of use, reliability, and performance.

Overview

Amazon Web Services (AWS) provides on-demand cloud computing platforms and APIs to individual companies. Each available AWS service generates information related to different aspects of its functionality. The available data types include service events, audit events, metrics, and logs.

...

Rw ui tabs macro
Rw tab
titleCloud collector

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

  1. In the Collector Server GUI, access the domain in which you want this instance to be created

  2. Click Add Collector and find the one you wish to add.

  3. In the Version field, select the latest value.

  4. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).

  5. In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.

  6. In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

Code Block
{
  "global_overrides": {
    "debug": false
  },
  "inputs": {
    "aws": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "access_key": "<access_key_value>",
        "access_secret": "<access_secret_value>",
        "base_assume_role": "<base_assume_role_value>",
        "target_assume_role": "<target_assume_role_value>",
        "assume_role_external_id": "<assume_role_external_id_value>"
      },
      "services": {
        "service-events-all": {
          "request_period_in_seconds": <request_period_in_seconds_value>,
          "cloudwatch_sqs_queue_name": "<sqs_queue_name_value>",
          "auto_event_type": <auto_event_type_value>,
          "regions": <list_of_regions>
        },
        "sqs-cloudwatch-consumer": {
          "request_period_in_seconds": <request_period_in_seconds_value>,
          "cloudwatch_sqs_queue_name": "<sqs_queue_name_value>",
          "auto_event_type": <auto_event_type_value>,
          "regions": <list_of_regions>
        },
        "audit-events-all": {
          "types": <list_of_types>,
          "request_period_in_seconds": <request_period_in_seconds_value>,
          "start_time": "<start_time_value>",
          "auto_event_type": <auto_event_type_value>,
          "drop_event_names": <list_of_drop_event_names>,
          "audit_sqs_queue_name": "<sqs_queue_name_value>",
          "s3_file_type_filter": "<s3_file_type_filter_value>",
          "use_region_and_account_id_from_event": "<use_region_and_account_id_from_event_value>",
          "regions": <list_of_regions>
        },
        "metrics-all":{
          "regions": <list_of_regions>
        },
        "non-cloudwatch-logs": {
          "types": <list_of_types>,
          "regions": <list_of_regions>,
          "start_time": "<start_time_value>",
          "vpcflowlogs_sqs_queue_name": "<sqs_queue_name_value>",
          "cloudfrontlogs_sqs_queue_name": "<sqs_queue_name_value>"
        },
        "cisco-umbrella": {
          "start_time": "<start_time_value>",
          "ciscoumbrella_sqs_queue_name": "<sqs_queue_name>",
          "regions": <list_of_regions>
        },
        "aws-guardduty": {
          "start_time": "<start_time_value>",
          "regions": [
            "ap-southeast-1"
          ]
        },
        "custom_service": {
          "types": "<list_of_types>",
          "log_group": "<log_group_value>",
          "start_time": "<start_time_value>",
          "regions": <list_of_regions>
        }
      }
    }
  }
}
Info

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Please replace the placeholders with real world values following the description table below:

Parameter

Data type

Type

Value range

Details

short_unique_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

access_key_value

str

See Accepted authentication methods section above.

Minimum length: 1

The access key ID value obtained from AWS when a user is created to access programmatically. It is used when authenticating with a user account and also to assume a self-account role.

access_secret_value

str

See Accepted authentication methods section above.

Minimum length: 1

The secret access key value obtained from AWS when a user is created to access programatically. It is used when authenticating with a user account and also to assume a self-account role.

base_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the base account. It can be used for self- or cross-account authentication methods.

target_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the customer’s account. It is used for cross-account authentication method.

assume_role_external_id_value

str

See Accepted authentication methods section above.

Minimum length: 1

This is an optional string implemented by the customer to add an extra security layer. It can only be used for cross-account authentication method.

request_period_in_seconds_value

int

Optional

Minimum length: 1

Period in seconds used between each data pulling, this value will overwrite the default value (300 seconds)

This parameter should be removed if it is not used.

auto_event_type_value

bool

Optional

true/false

Used to enable the auto categorization of message tagging.

start_time_value

datetime

Mandatory for GuardDuty, Optional for the rest of services.

1970-01-01T00:00:00.000Z

Date time from which to start collecting data. It must match ISO-8601 format. Note that this is mandatory for GuardDuty and optional for the rest of services.

list_of_types

list (of strings)

Optional

Code Block
"types" : [
  "type1",
  "type2",
  "type3"
]

Enable/Disable modules only when several modules per service are defined. For example, to get audit events from the API, this field should be set to audits_api.

list_of_regions

list (of strings)

Mandatory, if defined in the collector’s definition.

Code Block
"regions": [
  "region1",
  "region2",
  "region3"
]

Property name (regions) should be aligned with the one defined in the submodules_property property from the “Collector definitions”

list_of_drop_event_names

list (of strings)

Optional

Code Block
"drop_event_names" : [
  "drop1",
  "drop2",
  "drop3"
]

If the value in eventName field matches any of the values in this field, the event will be discarded.

i.e. if this parameter is populated with the next values ["Decrypt", "AssumeRole"], and the value of eventName field is Decrypt or AssumeRole, the event will be discarded.

sqs_queue_name_value

str

Mandatory

Minimum length: 1

Name of the SQS queue to read from.

s3_file_type_filter_value

str

Optional

Minimum length: 1

RegEx to retrieve proper file type from S3, in case there are more than one file types in the same SQS queue from which the service is pulling data.

This parameter can be used for those services getting data from a S3+SQS pipeline.

use_region_and_account_id_from_event_value

bool

Optional

true/false

If true the region and account_id are taken from the event; else if false, they are taken from the account used to do the data pulling.

Default: true

It can be used only in those services using a S3+SQS pipeline.

log_group_value

str

Mandatory

Minimum length: 1

The log group name must be set here as-is, including the different levels separated by slashes.
It can be set to'/' (forward slash) as well to get all the log group names.

AWS_10.png
Rw tab
titleOn-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block
<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml
Note

Replace <product_name> with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Note

Replace <product_name> with the proper value.

Editing the config.yaml file

Code Block
globals:
  debug: false
  id: <collector_id>
  name: <collector_name>
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  aws:
    id: <short_unique_id>
    enabled: true
    credentials:
      access_key: <access_key_value>
      access_secret: <access_secret_value>
      base_assume_role: <base_assume_role_value>
      target_assume_role: <target_assume_role_value>
      assume_role_external_id: <assume_role_external_id_value>
    services:
      service-events-all:
        request_period_in_seconds: <request_period_in_seconds_value>
        cloudwatch_sqs_queue_name: <sqs_queue_name_value>
        auto_event_type: <auto_event_type_value>
        regions: <list_of_regions>
      sqs-cloudwatch-consumer:
        request_period_in_seconds: <request_period_in_seconds_value>
        cloudwatch_sqs_queue_name: <sqs_queue_name_value>
        auto_event_type: <auto_event_type_value>
        regions: <list_of_regions>
      audit-events-all:
        types: <list_of_types>
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time: <start_time_value>
        auto_event_type: <auto_event_type_value>
        drop_event_names: <list_of_drop_event_names>
        audit_sqs_queue_name: <sqs_queue_name_value>
        s3_file_type_filter: <s3_file_type_filter_value>
        use_region_and_account_id_from_event: <use_region_and_account_id_from_event_value>
        regions: <list_of_regions>
      metrics-all:
        regions: <list_of_regions>
      non-cloudwatch-logs:
        types: <list_of_types>
        regions: <list_of_regions>
        start_time: <start_time_value>
        vpcflowlogs_sqs_queue_name: <sqs_queue_name_value>
        cloudfrontlogs_sqs_queue_name: <sqs_queue_name_value>
      cisco-umbrella:
        ciscoumbrella_sqs_queue_name: <sqs_queue_name_value>
        regions: <list_of_regions>
      custom_service:
        types: <list_of_types>
        log_groups: <log_group_value>
        start_time: <start_time_value>
        regions: <list_of_regions>
      aws-guardduty:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time: <start_time_value>
        regions: <list_of_regions>
Info

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Replace the placeholders with your required values following the description table below:

Parameter

Data Type

Type

Value Range

Details

collector_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this collector.

collector_name

str

Mandatory

Minimum length: 1
Maximum length: 10

Use this param to give a valid name to this collector.

devo_address

str

Mandatory

collector-us.devo.io
collector-eu.devo.io

Use this param to identify the Devo Cloud where the events will be sent.

chain_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the chain.cert  file downloaded from your Devo domain. Usually this file's name is: chain.crt

cert_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.cert downloaded from your Devo domain.

key_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.key downloaded from your Devo domain.

short_unique_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

access_key_value

str

See Accepted authentication methods section above.

Minimum length: 1

The access key ID value obtained from AWS when a user is created to access programmatically. It is used when authenticating with a user account and also to assume a self-account role.

access_secret_value

str

See Accepted authentication methods section above.

Minimum length: 1

The secret access key value obtained from AWS when a user is created to access programatically. It is used when authenticating with a user account and also to assume a self-account role.

base_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the base account. It can be used for self- or cross-account authentication methods.

target_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the customer’s account. It is used for cross-account authentication method.

assume_role_external_id_value

str

See Accepted authentication methods section above.

Minimum length: 1

This is an optional string implemented by the customer to add an extra security layer. It can only be used for cross-account authentication method.

request_period_in_seconds_value

int

Optional

Minimum length: 1

Period in seconds used between each data pulling, this value will overwrite the default value (300 seconds)

Info

This parameter should be removed if it is not used.

auto_event_type_value

bool

Optional

true/false

Used to enable the auto categorization of message tagging.

start_time_value

datetime

Mandatory for GuardDuty, Optional for the rest of services.

1970-01-01T00:00:00.000Z

Date time from which to start collecting data. It must match ISO-8601 format. Note that this is mandatory for GuardDuty and optional for the rest of services.

list_of_types

list (of strings)

Optional

Code Block
types:
  - type1
  - type2
  - type3

Enable/Disable modules only when several modules per service are defined. For example, to get audit events from the API, this field should be set to audits_api.

list_of_regions

list (of strings)

Mandatory, if defined in the collector’s definition.

Code Block
regions:
  - region1
  - region2
  - region3

Property name (regions) should be aligned with the one defined in the submodules_property property from the “Collector definitions”

list_of_drop_event_names

list (of strings)

Optional

Code Block
drop_event_names:
  - drop1
  - drop2
  - drop3

If the value in eventName field matches any of the values in this field, the event will be discarded.

i.e. if this parameter is populated with the next values ["Decrypt", "AssumeRole"], and the value of eventName field is Decrypt or AssumeRole, the event will be discarded.

sqs_queue_name_value

str

Mandatory

Minimum length: 1

Name of the SQS queue to read from.

s3_file_type_filter_value

str

Optional

Minimum length: 1

RegEx to retrieve proper file type from S3, in case there are more than one file types in the same SQS queue from which the service is pulling data.

This parameter can be used for those services getting data from a S3+SQS pipeline.

use_region_and_account_id_from_event_value

bool

Optional

true/false

If true the region and account_id are taken from the event; else if false, they are taken from the account used to do the data pulling.

Default: true

It can be used only in those services using a S3+SQS pipeline.

log_group_value

str

Mandatory

Minimum length: 1

The log group name must be set here as-is, including the different levels separated by slashes.
It can be set to'/' (forward slash) as well to get all the log group names.

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-aws-if-docker-image-1.1011.0

e5f62752b1fd91ca3347f7a639933bdd6db56d9f411b088352b737da0d1ee30872bcdf402ee5e9960ee8f5f26b4573b1107abf48263b0c494d48ce54408067ce

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz | docker load
Note

Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block
docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>
Note

Replace <product_name>, <image_name> and <version> with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block
version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note

Replace <product_name>, <image_name> and <version> with the proper values.

...

Release

Released on

Release type

Details

Recommendations

v1.11.0


Status
colourGreen
titleIMPROVEMENT

Status
colourRed
titleBUG FIX

Improvements

  • Updated DCSDK base docker image 1.3.1.

  • Added Unit tests and added user_guide

  • Upgraded Boto3 libraries from 1.34.97 to 1.35.92

  • Updated DCSDK from 1.11.1 to 1.13.1:

    • Added new sender for relay in house + TLS

    • Added persistence functionality for gzip sending buffer

    • Added Automatic activation of gzip sending

    • Improved behaviour when persistence fails

    • Upgraded DevoSDK dependency

    • Fixed console log encoding

    • Restructured python classes

    • Improved behaviour with non-utf8 characters

    • Decreased default size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • Applied changes to make DCSDK compatible with MacOS

    • Upgrade DevoSDK dependency to version v5.4.0

    • Change internal queue management for protecting against OOMK

    • Extracted ModuleThread structure from PullerAbstract

    • Improve Controlled stop when both processes fails to instantiate

    • Improve Controlled stop when InputProcess is killed

    • Bug related to lost of collector_name , collector_id and job_id

    • Bug related queues and ValueError (edited)

    • Change internal queue management for protecting against OOMK

    • Extracted ModuleThread structure from PullerAbstract

    • Improve Controlled stop when both processes fails to instantiate

    • Improve Controlled stop when InputProcess is killed

    • Fixed error related a ValueError exception not well controlled

    • Fixed error related with loss of some values in internal mes

  • Bug Fix:

    • Changes in code to handle the guard-duty missing logs issue

    sages

v1.10.0

Status
colourPurple
titleNEW FEATURE

Improvements:

  • Implemented GuardDuty service, added puller set-up and puller for it

Upgrade

v1.8.2

Status
colourGreen
titleIMPROVEMENT

Improvements:

  • Upgraded DCSDK Docker base image updated to 1.2.0

Upgrade

v1.8.1

Status
colourRed
titleBUG FIX

Bug Fixes:

  • Fix a bug when dealing with events that have no lastEventTimestamp present in the log_stream

Upgrade

v1.8.0

Status
colourGreen
titleIMPROVEMENT

Status
colourPurple
titleNEW FEATURE

New Feature

  • Updated method to call all the log group name if log_group parameter is this '/' in the config

Improvements

  • Upgraded DCSDK from 1.9.2 to 1.10.2

    • Ensure special characters are properly sent to the platform

    • Changed log level to some messages from info to debug

    • Changed some wrong log messages

    • Upgraded some internal dependencies

    • Changed queue passed to setup instance constructor

    • Ability to validate collector setup and exit without pulling any data

    • Ability to store in the persistence the messages that couldn't be sent after the collector stopped

    • Ability to send messages from the persistence when the collector starts and before the puller begins working

    • Ensure special characters are properly sent to the platform

Upgrade

v1.7.1

Status
colourRed
titlebug fixes

  • Fixed the way the collector handles milliseconds as the strptime function has been updated since 2021

  • Fixed the missing parameter in a method call

Recommended version

v1.6.0

Status
colourPurple
titleNEW FEATURE

New features:

  1. Added Cisco Umbrella new data source using SQS+S3

  2. Added is_aws_service optional parameter in collector_definitions.yaml.

  3. Added event_type_file_regex_patterns optional parameter to set a dict as: event_type -> regex_for_s3_file_key

Upgrade

v1.5.0

Status
colourGreen
titleIMPROVEMENT

Improvements

  1. Upgraded [boto] libraries from 1.21.36 to 1.28.24

  2. Upgraded DCSDK from 1.3.0 to 1.9.1

Upgrade

v1.4.1

Status
colourRed
titleBUG FIX

Bug Fixes:

  • Fixed a bug that prevented the use of the Assumed Role authentication method.

  • Fixed a bug that prevented session renewal when using any of the Assume Authentication methods:

    • Assume Role

    • Cross Account

Upgrade

v1.4.0

Status
colourPurple
titleNEW FEATURE

Status
colourGreen
titleIMPROVEMENT

Status
colourRed
titleBUG FIX

New features:

  • CrossAccount authentication method is now available improving the way in which the credentials are shared when the collector is running in the Collector Service.

Improvements:

  • The audit-events-all service (type audits_api) has been enhanced to allow requesting events older than 500 days.

Bug Fixes:

  • Fixed a bug that raised a KeyError when the optional param event_type_processor_mapping was not defined running service-events-all service.

Upgrade

...