We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer: In the Collector Server GUI, access the domain in which you want this instance to be created Click Add Collector and find the one you wish to add. In the Version field, select the latest value. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain). In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups. In the Parameters section, establish the Collector Parameters as follows below:
Editing the JSON configuration| Code Block |
|---|
{
"global_overrides": {
"debug": false
},
"inputs": {
"sqs_collector": {
"id": "12351",
"enabled": true,
"credentials": {
"aws_access_key_id": "",
"aws_secret_access_key": "",
"aws_base_account_role": "arn:aws:iam::837131528613:role/devo-xaccount-cs-role",
"aws_cross_account_role": "",
"aws_external_id": ""
},
"ack_messages": false,
"direct_mode": false,
"do_not_send": false,
"compressed_events": false,
"debug_md5": true,
"base_url": "https://us-west-1.queue.amazonaws.com/id/name-of-queue",
"region": "us-west-1",
"sqs_visibility_timeout": 240,
"sqs_wait_timeout": 20,
"sqs_max_messages": 1,
"services": {
"custom_service": {
"file_field_definitions": {},
"filename_filter_rules": [],
"encoding": "gzip",
"send_filtered_out_to_unknown": false,
"file_format": {
"type": "line_split_processor",
"config": {
"json": true
}
},
"record_field_mapping": {
"event_simpleName": {
"keys": [
"event_simpleName"
]
}
},
"routing_template": "edr.crowdstrike.cannon",
"line_filter_rules": [
[
{
"source": "record",
"key": "event_simpleName",
"type": "match",
"value": "EndOfProcess"
}
],
[
{
"source": "record",
"key": "event_simpleName",
"type": "match",
"value": "DeliverLocalFXToCloud"
}
]
]
}
}
}
}
} |
| Info |
|---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
| Note |
|---|
Please replace the placeholders with real world values following the description table below |
Parameter | Data type | Type | Value range / Format | Details | debug_status
| bool
| Mandatory
| false / true
| If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed. | short_unique_id
| int
| Mandatory
| Minimum length: 1
Maximum length: 5 | Use this param to give an unique id to this input service. | Note |
|---|
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. |
| enabled
| bool
| Mandatory
| false / true
| Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored. | base_url
| str
| Mandatory
| | By default, the base url is https://sqs.region.amazonaws.com/account-number/queue-name. This needs to be set to the url of sqs. | aws_access_key_id
| str
| Mandatory/Optional
| Any | Only needed if not using cross account | aws_secret_access_key
| str
| Mandatory/Optional
| Any | Only needed if not using cross account | aws_base_account_role
| str
| Mandatory/Optional
| Any | Only needed if using cross account This is devos cross account role | aws_cross_account_role
| str
| Mandatory/Optional
| Any | Only needed if using cross account This is your cross account role | aws_external_id
| str
| Optional
| Any | Extra security you can set up | ack_messages
| bool
| Manatory
| false / true
| Needs to be set to true to delete messages from the queue. Leave false until testing complete | direct_mode
| bool
| Optional
| false / true
| Set to False for most all scenarios. This parameter should be removed if it is not used. | do_not_send
| bool
| Optional
| false / true
| Set to True to not send the log to Devo. This parameter should be removed if it is not used. | debug_md5
| bool
| Optional
| false / true
| Set to True to will send the message md5 to my.app.sqs.message_body only needed for more debugging on duplicates. This parameter should be removed if it is not used. | sqs_visibility_timeout
| int
| Mandatory
| Min: 120 Max: 43200 (haven’t needed to test higher) | This parameter specifies how long the object will be held by the collector. If it is not processed and deleted within the allotted time in seconds. The message will be put back and can be processed again. Set this parameter for timeouts between the queue and the collector, the collector has to download large files and process them. If this process is broken up the time. Otherwise defaults to 120. For Crowdstrike FDR some messages can take 10-15 minutes to process please set the timeout to help duplicate reduction. | sqs_wait_timeout
| int
| Mandatory
| Min: 20 Max: 20 | The min has handled most customer scenarios at this point. | sqs_max_messagesThis is how long polling works. It will wait per poll the value of seconds listed. If no message is found, it will return Long poll did not find any messages in queue. All data in the SQS queue has been successfully collected.
| sqs_max_messages
| int
| Mandatory
| Min: 1 Max: 6 | This is now 1 always and forever. | region
| str
| Mandatory
| Example: us-east-1
| This is the region that is in the base url | compressed_events
| bool
| Mandatory
| This needs to be true or False | Only works with GZIP compression should be false unless you see this below. If you see any errors ‘utf-8' codec can't decode byte 0xa9 in position 36561456: invalid start byte it might be the events need to be decompressed |
| Rw tab |
|---|
| title | On-premise collector |
|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector: | Code Block |
|---|
<any_directory>
└── devo-collectors/
└── <product_name>/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── state/
└── config/
└── config.yaml |
| Note |
|---|
Replace <product_name> with the proper value. |
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.  | Note |
|---|
Replace <product_name> with the proper value. |
Editing the config.yaml file| Code Block |
|---|
globals:
debug: <debug_status>
id: <collector_id>
name: <collector_name>
persistence:
type: filesystem
config:
directory_name: state
multiprocessing: false
queue_max_size_in_mb: 1024
queue_max_size_in_messages: 1000
queue_max_elapsed_time_in_sec: 60
queue_wrap_max_size_in_messages: 100
outputs:
devo_1:
type: devo_platform
config:
address: <devo_address>
port: 443
type: SSL
chain: <chain_filename>
cert: <cert_filename>
key: <key_filename>
inputs:
sqs:
id: 12345
enabled: true
credentials:
aws_access_key_id: password
aws_secret_access_key: secret-access-key
aws_base_account_role: arn:aws:iam::837131528613:role/devo-xaccount-cs-role
aws_cross_account_role: arn:aws:iam::{account-id}:role/{role-name}
aws_external_id: extra_security_optional
region: region
base_url: https://sqs.{region}.amazonaws.com/{account-number}/{queue-name}
sqs_visibility_timeout: 120
sqs_wait_timeout: 20
sqs_max_messages: 4
ack_messages: false
direct_mode: false
do_not_send: false
compressed_events: false
services:
custom_service:
file_field_definitions: {}
filename_filter_rules: []
encoding: gzip
ack_messages: false
file_format:
type: single_json_object_processor
config:
key: Records
record_field_mapping: {}
routing_template: my.app.source1.type1
line_filter_rules: [] |
| Info |
|---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Replace the placeholders with your required values following the description table below: Parameter | Data type | Type | Value range | Details | debug_status
| bool
| Mandatory
| false / true
| If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed. | collector_id
| int
| Mandatory
| Minimum length: 1
Maximum length: 5 | Use this param to give an unique id to this collector. | collector_name
| str
| Mandatory
| Minimum length: 1
Maximum length: 10 | Use this param to give a valid name to this collector. | devo_address
| str
| Mandatory
| collector-us.devo.io
collector-eu.devo.io
| Use this param to identify the Devo Cloud where the events will be sent. | chain_filename
| str
| Mandatory
| Minimum length: 4
Maximum length: 20 | Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt | cert_filename
| str
| Mandatory
| Minimum length: 4
Maximum length: 20 | Use this param to identify the file.cert downloaded from your Devo domain. | key_filename
| str
| Mandatory
| Minimum length: 4
Maximum length: 20 | Use this param to identify the file.key downloaded from your Devo domain. | short_unique_id
| int
| Mandatory
| Minimum length: 1
Maximum length: 5 | Use this param to give an unique id to this input service. | Note |
|---|
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. |
| input_status
| bool
| Mandatory
| false / true
| Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored. | base_url
| str
| Mandatory
| | By default, the base url is https://sqs.region.amazonaws.com/account-number/queue-name. This needs to be set to the url of sqs. | aws_access_key_id
| str
| Mandatory/Optional
| Any | Only needed if not using cross account | aws_secret_access_key
| str
| Mandatory/Optional
| Any | Only needed if not using cross account | aws_base_account_role
| str
| Mandatory/Optional
| Any | Only needed if using cross account This is devos cross account role | aws_cross_account_role
| str
| Mandatory/Optional
| Any | Only needed if using cross account This is your cross account role | aws_external_id
| str
| Optional
| Any | Extra security you can set up | ack_messages
| bool
| Manatory
| false / true
| Needs to be set to true to delete messages from the queue. Leave false until testing complete | direct_mode
| bool
| Optional
| false / true
| Set to False for most all scenarios. This parameter should be removed if it is not used. | do_not_send
| bool
| Optional
| false / true
| Set to True to not send the log to Devo. This parameter should be removed if it is not used. | debug_md5
| bool
| Optional
| false / true
| Set to True to will send the message md5 to my.app.sqs.message_body only needed for more debugging on duplicates. This parameter should be removed if it is not used. | sqs_visibility_timeout
| int
| Mandatory
| Min: 120 Max: 43200 (haven’t needed to test higher) | This parameter specifies how long the object will be held by the collector. If it is not processed and deleted within the allotted time in seconds. The message will be put back and can be processed again. Set this parameter for timeouts between the queue and the collector, the collector has to download large files and process them. If this process is broken up the time. Otherwise defaults to 120. sqs_For Crowdstrike FDR some messages can take 10-15 minutes to process please set the timeout to help duplicate reduction. | sqs_wait_timeout
| int
| Mandatory
| Min: 20 Max: 20The | min has handled most customer scenarios at this pointThis is how long polling works. It will wait per poll the value of seconds listed. If no message is found, it will return Long poll did not find any messages in queue. All data in the SQS queue has been successfully collected. | sqs_max_messages
| int
| Mandatory
| Min: 1 Max: 6 | This is now 1 always and forever. | region
| str
| Mandatory
| Example: us-east-1
| This is the region that is in the base url | compressed_events
| bool
| Mandatory
| This needs to be true or False | Only works with GZIP compression should be false unless you see this below. If you see any errors ‘utf-8' codec can't decode byte 0xa9 in position 36561456: invalid start byte it might be the events need to be decompressed |
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: Collector Docker image | SHA-256 hash |
|---|
collector-aws_sqs_if-docker-image-1.6.14 | 916e5adbebc9241b79577d088320e4f940feab35dfbc160070380fff565af03b0f086c864b27446f864679476ed167f0c69f306f19bd735bffeb94d1a477f9c2
|
Use the following command to add the Docker image to the system: | Code Block |
|---|
gunzip -c <image_file>-<version>.tgz | docker load |
| Note |
|---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value. |
The Docker image can be deployed on the following services: DockerExecute the following command on the root directory <any_directory>/devo-collectors/<product_name>/ | Code Block |
|---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
| Note |
|---|
Replace <product_name>, <image_name> and <version> with the proper values. |
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory. | Code Block |
|---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory: | Code Block |
|---|
IMAGE_VERSION=<version> docker-compose up -d |
| Note |
|---|
Replace <product_name>, <image_name> and <version> with the proper values. |
|
