...
Eliminate blind spots with complete visibility
Simplify security operations to cut mean time to respond (MTTR)
Harness the scale of the cloud for AI and analytics
Lower costs by consolidating tools and improving SOC efficiency.
Cortex XDR collector migration guide (from 1.x.x to 2.x.x)
If you need to migrate an old collector version to a more recent one, please check the migration process in this article.
Devo collector features
Feature | Details |
|---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Requires IP Whitelisting |
|
...
| Rw ui tabs macro | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
The Collector Server is a managed platform that allows running sets of different collectors grouped by Devo domain destinations. To run an instance of this data collector, the next steps must be followed:
| ||||||||||||
Parameter | Data type | Type | Value range / Format | Details | ||||||||
|
|
|
| Use this param to give a unique id to this input service.
| ||||||||
|
|
|
| Use this param to enable or disable the given input logic when running the collector. If the value is |
|
|
| Minimum Length
,
"audit_managements": {
"api_fqdn": "<api_fqdn>",
"request_period_in_seconds" : <request_period_in_seconds>",
"start_time": "<start_time>",
"override_devo_tag": "<override_devo_tag>"
}
}
}
}
} |
| Note |
|---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the |
| Info |
|---|
Please replace the placeholders with real world values following the description table below |
Parameter | Data type | Type | Value range / Format | Details | ||
|---|---|---|---|---|---|---|
|
|
|
| Use this param to give a unique id to this input service.
| ||
|
|
|
| Use this param to enable or disable the given input logic when running the collector. If the value is | ||
|
|
|
| The API Key is your unique identifier used as the | ||
|
|
|
| The API Key ID is your unique token used to authenticate the API Key. It is used in headers as | ||
|
|
|
| The FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: | ||
|
|
|
| Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds | ||
|
|
| A devo tag | This parameter allows to define a custom devo tag. | ||
|
|
| A Incident Alert tag | This Tag is only applicable for Incidents service to override the tag of Incident alerts( Extra incident endpoint). Ex: | ||
|
|
| A boolean to Include Incidents alerts | By default the value of this boolean is ‘true’. If given ‘alsefalse’ we will not be able to get incident alerts data (Extra incidents data for endpoint: | ||
|
|
| start time in utc | This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z |
| Rw tab | ||
|---|---|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
| Code Block |
|---|
<any_directory>
└── devo-collectors/
└── <product_name>/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── state/
└── config/
└── config.yaml |
| Note |
|---|
Replace |
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.
| Note |
|---|
Replace |
Editing the config.yaml file
| Code Block |
|---|
globals:
debug: false
id: not_used
name: cortex_xdr
persistence:
type: filesystem
config:
directory_name: state
outputs:
devo_1:
type: devo_platform
config:
address: collector-us.devo.io
port: 443
type: SSL
chain: chain.crt
cert: <devo_domain>.crt
key: <devo_domain>.key
console_1:
type: console
inputs:
cortex_xdr:
id: <short_unique_id>
enabled: true
credentials:
api_key: <api_key>
api_key_id: <api_key_id>
services:
incidents:
api_fqdn: <api_fqdn>
request_period_in_seconds : <request_period_in_seconds> #optional
start_time: <start_time> #optional
include_incident_alerts: <include_incident_alerts> #Optional
override_devo_tag: <override_devo_tag> #optional
override_incident_alert_tag: <override_incident_alert_tag> #optional
alerts:
api_fqdn: <api_fqdn>
start_time: <start_time> # Example 2024-01-01T01:50:00Z
request_period_in_seconds: <request_period_in_seconds> #optional
override_devo_tag: <override_devo_tag> #optional
all_alerts:
api_fqdn: <api_fqdn>
start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
request_period_in_seconds: <opt_request_period_in_seconds> #optional
override_devo_tag: <override_devo_tag> #optional
audit_managements:
api_fqdn: <api_fqdn>
start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
request_period_in_seconds: <opt_request_period_in_seconds> #optional
override_devo_tag: <override_devo_tag> #optional
violations:
api_fqdn: <api_fqdn>
start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
request_period_in_seconds: <opt_request_period_in_seconds> #optional
override_devo_tag: <override_devo_tag> #optional |
| Note |
|---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the |
| Info |
|---|
Please replace the placeholders with real world values following the description table below |
Parameter | Data type | Type | Value range / Format | Details | ||
|---|---|---|---|---|---|---|
|
|
|
| Use this param to give a unique id to this input service.
| ||
|
|
|
| Use this param to enable or disable the given input logic when running the collector. If the value is | ||
|
|
|
| The API Key is your unique identifier used as the | ||
|
|
|
| The API Key ID is your unique token used to authenticate the API Key. It is used in headers as | ||
|
|
|
| The {api_fqdn} FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: | ||
|
|
|
| Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds | ||
|
|
| A devo tag | This parameter allows to define a custom devo tag. | ||
|
|
| A Incident Alert tag | This Tag is only applicable for Incidents service to override the tag of Incident alerts ( Extra incident endpoint). Ex: | ||
|
|
| A boolean to Include Incidents alerts | By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data (Extra incidents data for endpoint: | ||
|
|
| start time in utc | This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z |
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Collector Docker image | SHA-256 hash |
|---|---|
|
Use the following command to add the Docker image to the system:
| Code Block |
|---|
gunzip -c <image_file>-<version>.tgz | docker load |
| Note |
|---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace |
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
| Code Block |
|---|
docker run --name collector-<product_name> --volume $PWD/certs:/devo-collector/certs --volume $PWD/config:/devo-collector/config --volume $PWD/state:/devo-collector/state --env CONFIG_FILE=config.yaml --rm --interactive --tty <image_name>:<version> |
| Note |
|---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.
| Code Block |
|---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:
| Code Block |
|---|
IMAGE_VERSION=<version> docker-compose up -d |
| Note |
|---|
Replace |
...
| Expand | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
Verify data collectionService componentsOnce the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components:
Setup is common for every service, so its output is displayed in this section. On the other hand, Pullers are specific to each Service, so their outputs are shown in their corresponding sections below. Setup outputA successful run has the following output messages for the setup module:
Restart the persistenceSome services in this collector use persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
|
| Expand | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
Restart the persistenceSome services in this collector use persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
| ||||||||
| Expand | ||||||||
| ||||||||
Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. |
| Expand | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. | ||||||||
| Expand | ||||||||
| ||||||||
Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. |
| Expand | ||||||
|---|---|---|---|---|---|---|
| ||||||
Devo categorization and destinationAll events of this service are ingested into table Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. | ||||||
| Expand | ||||||
| ||||||
Devo categorization and destinationAll events of this service are ingested into table Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. |
| Expand | ||||||
|---|---|---|---|---|---|---|
| ||||||
Devo categorization and destinationAll events of this service are ingested into table Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. | ||||||
| Expand | ||||||
| ||||||
Devo categorization and destinationAll events of this service are ingested into table Verify data collectionPuller outputA successful initial run has the following output messages for the puller module: Note that the
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. |
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
| Expand | ||
|---|---|---|
| ||
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. |
| Expand | ||||||
|---|---|---|---|---|---|---|
| ||||||
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module, including all services together:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistenceThis service makes use of persistence. You can check how to restart it above. |
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
| Expand | ||
|---|---|---|
| ||
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. |
| Expand | ||
|---|---|---|
| ||
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module, including all services together:
|
| Expand | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module:
| ||||||||||
| Expand | ||||||||||
| ||||||||||
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module:
By default, these information traces will be displayed every 10 minutes. Sender statisticsEach service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
|
| Expand | ||||
|---|---|---|---|---|
| ||||
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
By default, these information traces will be displayed every 10 minutes. Sender statisticsEach service displays its own performance statistics that allow checking how many events have been delivered to Devo by type: | ||||
Logging trace | Description | |||
| Displays the number of concurrent senders available for the given Sender Service. | |||
| Displays the items available in the internal sender queue. This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. | |||
| Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
By default these traces will be shown every 10 minutes. |
| Expand | ||||
|---|---|---|---|---|
| ||||
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
Differences between
|
| Expand | ||
|---|---|---|
| ||
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode. |
...
| title | Troubleshooting |
|---|
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.
...
Error type
...
Error ID
...
Error message
...
Cause
...
Solution
...
SetupError
...
100
...
HTTP Error occurred while checking the server health for cortex
...
If 401 client error, credentials used are not valid.
or issue with the API request process.
...
Ensure correct credentials are used if 401 client error. Else Contact devo support team
...
101
...
Some error occurred while checking the server health for cortex. Error details
...
The authentication was correct, but returned an unexpected status code.
...
This is an internal issue. Contact with Devo Support team.
...
PullError
...
300
...
Expected 200 status code, received {status code}
...
There has been an error upon API request process.
...
This is an internal issue. Contact with Devo Support team.
...
301
...
Expected 200 status code, received {status code}
...
You will get the status code with error msg.
...
Kindly reach the develop with the exact msg.
Change log
...
Release
...
Released on
...
Release type
...
Details
...
Differences between
|
| Expand | ||
|---|---|---|
| ||
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode. |
| Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.
|
Change log
Release | Released on | Release type | Recommendations | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
| ||||||||||||||
| ||||||||||||||||
|
|
| ||||||||||||||
| ||||||||||||||||
|
|
| ||||||||||||||
| ||||||||||||||||
|
|
| ||||||||||||||
| bug fixing
|
|
| |||||||||||||
|
|
| ||||||||||||||
| ||||||||||||||||
|
|
| ||||||||||||||
|
| |||||||||||||||
|
| | ||||||||||||||
| ||||||||||||||||