Page Comparison

...

Rw ui tabs macro

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

In the Collector Server GUI, access the domain in which you want this instance to be created
Click Add Collector and find the one you wish to add.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.
In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

Code Block

{
  "global_overrides": {
    "debug": false
  },
  "inputs": {
    "aws": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "access_key": "<access_key_value>",
        "access_secret": "<access_secret_value>",
        "base_assume_role": "<base_assume_role_value>",
        "target_assume_role": "<target_assume_role_value>",
        "assume_role_external_id": "<assume_role_external_id_value>"
      },
      "services": {
        "service-events-all": {
          "request_period_in_seconds": <request_period_in_seconds_value>,
          "cloudwatch_sqs_queue_name": "<sqs_queue_name_value>",
          "auto_event_type": <auto_event_type_value>,
          "regions": <list_of_regions>
        },
        "sqs-cloudwatch-consumer": {
          "request_period_in_seconds": <request_period_in_seconds_value>,
          "cloudwatch_sqs_queue_name": "<sqs_queue_name_value>",
          "auto_event_type": <auto_event_type_value>,
          "regions": <list_of_regions>
        },
        "audit-events-all": {
          "types": <list_of_types>,
          "request_period_in_seconds": <request_period_in_seconds_value>,
          "start_time": "<start_time_value>",
          "auto_event_type": <auto_event_type_value>,
          "drop_event_names": <list_of_drop_event_names>,
          "audit_sqs_queue_name": "<sqs_queue_name_value>",
          "s3_file_type_filter": "<s3_file_type_filter_value>",
          "use_region_and_account_id_from_event": "<use_region_and_account_id_from_event_value>",
          "regions": <list_of_regions>
        },
        "metrics-all":{
          "regions": <list_of_regions>
        },
        "non-cloudwatch-logs": {
          "types": <list_of_types>,
          "regions": <list_of_regions>,
          "start_time": "<start_time_value>",
          "vpcflowlogs_sqs_queue_name": "<sqs_queue_name_value>",
          "cloudfrontlogs_sqs_queue_name": "<sqs_queue_name_value>"
        },
        "cisco-umbrella": {
          "start_time": "<start_time_value>",
          "ciscoumbrella_sqs_queue_name": "<sqs_queue_name>",
          "regions": <list_of_regions>
        },
        "aws-guardduty": {
          "start_time": "<start_time_value>",
          "regions": [
            "ap-southeast-1"
          ]
        },
        "custom_service": {
          "types": "<list_of_types>",
          "log_group": "<log_group_value>",
          "start_time": "<start_time_value>",
          "regions": <list_of_regions>
        }
      }
    }
  }
}

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Please replace the placeholders with real world values following the description table below:

Parameter

Data type

Type

Value range

Details

short_unique_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

access_key_value

str

See Accepted authentication methods section above.

Minimum length: 1

The access key ID value obtained from AWS when a user is created to access programmatically. It is used when authenticating with a user account and also to assume a self-account role.

access_secret_value

str

See Accepted authentication methods section above.

Minimum length: 1

The secret access key value obtained from AWS when a user is created to access programatically. It is used when authenticating with a user account and also to assume a self-account role.

base_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the base account. It can be used for self- or cross-account authentication methods.

target_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the customer’s account. It is used for cross-account authentication method.

assume_role_external_id_value

str

See Accepted authentication methods section above.

Minimum length: 1

This is an optional string implemented by the customer to add an extra security layer. It can only be used for cross-account authentication method.

request_period_in_seconds_value

int

Optional

Minimum length: 1

Period in seconds used between each data pulling, this value will overwrite the default value (300 seconds)

This parameter should be removed if it is not used.

auto_event_type_value

bool

Optional

true/false

Used to enable the auto categorization of message tagging.

start_time_value

datetime

Mandatory for GuardDuty, Optional for the rest of services.

1970-01-01T00:00:00.000Z

Date time from which to start collecting data. It must match ISO-8601 format. Note that this is mandatory for GuardDuty and optional for the rest of services.

list_of_types

list (of strings)

Optional

Code Block
"types" : [ "type1", "type2", "type3" ]

Enable/Disable modules only when several modules per service are defined. For example, to get audit events from the API, this field should be set to audits_api.

list_of_regions

list (of strings)

Mandatory, if defined in the collector’s definition.

Code Block
"regions": [ "region1", "region2", "region3" ]

Property name (regions) should be aligned with the one defined in the submodules_property property from the “Collector definitions”

list_of_drop_event_names

list (of strings)

Optional

Code Block
"drop_event_names" : [ "drop1", "drop2", "drop3" ]

If the value in eventName field matches any of the values in this field, the event will be discarded.

i.e. if this parameter is populated with the next values ["Decrypt", "AssumeRole"], and the value of eventName field is Decrypt or AssumeRole, the event will be discarded.

sqs_queue_name_value

str

Mandatory

Minimum length: 1

Name of the SQS queue to read from.

s3_file_type_filter_value

str

Optional

Minimum length: 1

RegEx to retrieve proper file type from S3, in case there are more than one file types in the same SQS queue from which the service is pulling data.

This parameter can be used for those services getting data from a S3+SQS pipeline.

use_region_and_account_id_from_event_value

bool

Optional

true/false

If true the region and account_id are taken from the event; else if false, they are taken from the account used to do the data pulling.

Default: true

It can be used only in those services using a S3+SQS pipeline.

log_group_value

str

Mandatory

Minimum length: 1

The log group name must be set here as-is, including the different levels separated by slashes.
It can be set to'/' (forward slash) as well to get all the log group names.

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

globals:
  debug: false
  id: <collector_id>
  name: <collector_name>
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  aws:
    id: <short_unique_id>
    enabled: true
    credentials:
      access_key: <access_key_value>
      access_secret: <access_secret_value>
      base_assume_role: <base_assume_role_value>
      target_assume_role: <target_assume_role_value>
      assume_role_external_id: <assume_role_external_id_value>
    services:
      service-events-all:
        request_period_in_seconds: <request_period_in_seconds_value>
        cloudwatch_sqs_queue_name: <sqs_queue_name_value>
        auto_event_type: <auto_event_type_value>
        regions: <list_of_regions>
      sqs-cloudwatch-consumer:
        request_period_in_seconds: <request_period_in_seconds_value>
        cloudwatch_sqs_queue_name: <sqs_queue_name_value>
        auto_event_type: <auto_event_type_value>
        regions: <list_of_regions>
      audit-events-all:
        types: <list_of_types>
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time: <start_time_value>
        auto_event_type: <auto_event_type_value>
        drop_event_names: <list_of_drop_event_names>
        audit_sqs_queue_name: <sqs_queue_name_value>
        s3_file_type_filter: <s3_file_type_filter_value>
        use_region_and_account_id_from_event: <use_region_and_account_id_from_event_value>
        regions: <list_of_regions>
      metrics-all:
        regions: <list_of_regions>
      non-cloudwatch-logs:
        types: <list_of_types>
        regions: <list_of_regions>
        start_time: <start_time_value>
        vpcflowlogs_sqs_queue_name: <sqs_queue_name_value>
        cloudfrontlogs_sqs_queue_name: <sqs_queue_name_value>
      cisco-umbrella:
        ciscoumbrella_sqs_queue_name: <sqs_queue_name_value>
        regions: <list_of_regions>
      custom_service:
        types: <list_of_types>
        log_groups: <log_group_value>
        start_time: <start_time_value>
        regions: <list_of_regions>
      aws-guardduty:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time: <start_time_value>
        regions: <list_of_regions>

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Replace the placeholders with your required values following the description table below:

Parameter

Data Type

Type

Value Range

Details

collector_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this collector.

collector_name

str

Mandatory

Minimum length: 1
Maximum length: 10

Use this param to give a valid name to this collector.

devo_address

str

Mandatory

collector-us.devo.io
collector-eu.devo.io

Use this param to identify the Devo Cloud where the events will be sent.

chain_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt

cert_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.cert downloaded from your Devo domain.

key_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.key downloaded from your Devo domain.

short_unique_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

access_key_value

str

See Accepted authentication methods section above.

Minimum length: 1

The access key ID value obtained from AWS when a user is created to access programmatically. It is used when authenticating with a user account and also to assume a self-account role.

access_secret_value

str

See Accepted authentication methods section above.

Minimum length: 1

The secret access key value obtained from AWS when a user is created to access programatically. It is used when authenticating with a user account and also to assume a self-account role.

base_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the base account. It can be used for self- or cross-account authentication methods.

target_assume_role_value

str

See Accepted authentication methods section above.

Minimum length: 1

The ARN of the role to be assumed in the customer’s account. It is used for cross-account authentication method.

assume_role_external_id_value

str

See Accepted authentication methods section above.

Minimum length: 1

This is an optional string implemented by the customer to add an extra security layer. It can only be used for cross-account authentication method.

request_period_in_seconds_value

int

Optional

Minimum length: 1

Period in seconds used between each data pulling, this value will overwrite the default value (300 seconds)

Info
This parameter should be removed if it is not used.

auto_event_type_value

bool

Optional

true/false

Used to enable the auto categorization of message tagging.

start_time_value

datetime

Mandatory for GuardDuty, Optional for the rest of services.

1970-01-01T00:00:00.000Z

Date time from which to start collecting data. It must match ISO-8601 format. Note that this is mandatory for GuardDuty and optional for the rest of services.

list_of_types

list (of strings)

Optional

Code Block
types: - type1 - type2 - type3

Enable/Disable modules only when several modules per service are defined. For example, to get audit events from the API, this field should be set to audits_api.

list_of_regions

list (of strings)

Mandatory, if defined in the collector’s definition.

Code Block
regions: - region1 - region2 - region3

Property name (regions) should be aligned with the one defined in the submodules_property property from the “Collector definitions”

list_of_drop_event_names

list (of strings)

Optional

Code Block
drop_event_names: - drop1 - drop2 - drop3

If the value in eventName field matches any of the values in this field, the event will be discarded.

i.e. if this parameter is populated with the next values ["Decrypt", "AssumeRole"], and the value of eventName field is Decrypt or AssumeRole, the event will be discarded.

sqs_queue_name_value

str

Mandatory

Minimum length: 1

Name of the SQS queue to read from.

s3_file_type_filter_value

str

Optional

Minimum length: 1

RegEx to retrieve proper file type from S3, in case there are more than one file types in the same SQS queue from which the service is pulling data.

This parameter can be used for those services getting data from a S3+SQS pipeline.

use_region_and_account_id_from_event_value

bool

Optional

true/false

If true the region and account_id are taken from the event; else if false, they are taken from the account used to do the data pulling.

Default: true

It can be used only in those services using a S3+SQS pipeline.

log_group_value

str

Mandatory

Minimum length: 1

The log group name must be set here as-is, including the different levels separated by slashes.
It can be set to'/' (forward slash) as well to get all the log group names.

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-aws-if-docker-image-1.11.0	`a3c6395d62457e373e6104c1f68b8afb4205109572bcdf402ee5e9960ee8f5f26b4573b1107abf48263b0c494d48ce54408067ce`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

...

Version	Old Version 47	New Version 48
Changes made by	MD Tausif	MD Tausif
Saved on	Jan 10, 2025	Jan 10, 2025

Versions Compared

Key

Editing the JSON configuration

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose