Content Comparison

...

short_unique_

Rw ui tabs macro

Rw tab

title	Cloud collector

The Collector Server is a managed platform that allows running sets of different collectors grouped by Devo domain destinations.

To run an instance of this data collector, the next steps must be followed:

In the Collector Server GUI, access the domain where you want to create this instance, click Add Collector, search for “Cortex XDR - Integrations Factory”, then click on the result.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the Parameters section, establish the Collector Parameters as follows below:Collector services detail

Info
Please, replace the placeholders `<api_key_value>,` `<api_key_id_value>`, and `<api_fqdn_value>` in the next section with the values obtained in previous sections of this document, except the `<short_unique_identifier>` that can have the value you choose. Do not substitute the occurrences of `{api_fqdn}`.

Code Block

{
  "global_overrides": {
    "debug": <debug_status>
  },
  "inputs": {
    "cortex_xdr": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "api_key": "<api_key>",
        "api_key_id": "<api_key_id>"
      },
      "services": {
        "incidents": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "include_incident_alerts": "<include_incident_alerts>",
          "override_devo_tag": "<override_devo_tag>"
          "override_incident_alert_tag": "<override_incident_alert_tag>"
        },
        "alerts": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "all_alerts": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "violations": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "audit_managements": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        }
      }
    }
  }
}

Note
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Info
Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

<short_unique_id>

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

<input_status>

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

<api_key>

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

<api_key_id>

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id}

<api_fqdn>

str

Mandatory

Minimum Length 1

The FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

<request_period_in_seconds>

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

<override_devo_tag>

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

<override_incident_alert_tag>

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts( Extra incident endpoint). Ex: my.app.devo.Incident_alert

<include_incident_alerts>

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘alsefalse’ we will not be able to get incident alerts data

(Extra incidents data for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

<start_time>

str

Optional

Info
Note: For This endpoint, the data gets ingested in the table `edr.cortex_xdr.incident_alert` and this table may contains duplicates because we are using `incident_id` to get the data and these `incident_ids` can be duplicates. We are using `modification_time` instead of `creation_time` to filter the `incident_ids` causing duplicates.

<start_time>

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

globals:
  debug: false
  id: not_used
  name: cortex_xdr
  persistence:
    type: filesystem
    config:
      directory_name: state

outputs:
  devo_1:
    type: devo_platform
    config:
      address: collector-us.devo.io
      port: 443
      type: SSL
      chain: chain.crt
      cert: <devo_domain>.crt
      key: <devo_domain>.key
  console_1:
    type: console

inputs:
  cortex_xdr:
    id: <short_unique_id>
    enabled: true
    credentials:
      api_key: <api_key>
      api_key_id: <api_key_id>
    services:
      incidents:
        api_fqdn: <api_fqdn>
        request_period_in_seconds : <request_period_in_seconds> #optional
        start_time: <start_time> #optional
        include_incident_alerts: <include_incident_alerts> #Optional
        override_devo_tag: <override_devo_tag> #optional
        override_incident_alert_tag: <override_incident_alert_tag> #optional
      alerts:
        api_fqdn: <api_fqdn>
        start_time: <start_time> # Example 2024-01-01T01:50:00Z
        request_period_in_seconds: <request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      all_alerts:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      audit_managements:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      violations:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional

Note
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Info
Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

Note
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Info
Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

short_unique_id

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

api_key

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

api_key_id

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id} while calling the API.

api_fqdn

str

Mandatory

Minimum Length 1

The {api_fqdn} FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

request_period_in_seconds

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

override_devo_tag

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

override_incident_alert_tag

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts ( Extra incident endpoint). Ex: my.app.devo.Incident_alert

include_incident_alerts

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data (Extra incidents data for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

start_time

str

Info
Note: For This endpoint, the data gets ingested in the table `edr.cortex_xdr.incident_alert` and this table may contains duplicates because we are using `incident_id` to get the data and these `incident_ids` can be duplicates. We are using `modification_time` instead of `creation_time` to filter the `incident_ids` causing duplicates.

start_time

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-cortex_xdr_if-docker-image-2.0.14	`1f7a3dd54371e26538b35fced4a8965bb7534ce7f84360065fc815711c856cb86a1c62cfeb366873d2fcc67dbb2ccefc69841941e99b054f1ab8b27aca8752f5`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

...

Expand

title	Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.

The authentication was correct, but returned an unexpected status code

Error type	Error ID	Error message	Cause	Solution
`SetupError`	100	`HTTP Error occurred while checking the server health for cortex`	If 401 client error, credentials used are not valid. or issue with the API request process.	Ensure correct credentials are used if 401 client error. Else Contact devo support team	101	`Some error occurred while checking the server health for cortex. Error details`	`from remote source: {e}`	You will get the status code with error msg.	Ensure that the collector has the necessary permissions and proper credentials to access the Cortex xdr API and contact the developer with exact error message
`PullError`	300	`Error while fetching the event data for service {service_name} and error: {e}`	You will get the status code with error msg.	This is an internal issue. Contact with Devo Support team.
`PullError`	300	`Expected 200 status code, received {status code}`	There has been an error upon API request process.	This is an internal issue. Contact with Devo Support team.
`PullError`	301	`Expected 200 status code, received {status code}`	You will get the status code with error msg.	Kindly reach the develop with the exact msg.

Change log

...

Release

...

Released on

...

Release type

...


`ApiError`	400	`HTTP ERROR 400: Bad request: The server could not understand the request`	Bad Request	Kindly reach out to the developer
	401	`HTTP ERROR 401: Unauthorized: Authentication is required and has failed or has not been provided`	Credentials provided is not correct	Please check and provide the correct credentials
	402	`Unauthorized access. User does not have the required license type to run this API`	User doesn't have the required permission	Kindly get required permission and reach the developer with the exact msg.
	403	`HTTP ERROR 500: Server Error: An error occurred on the server`	Internal server error. An unverified status of API communication type error	Kindly reach the developer with the exact msg.
	404	`Request Error: Received status code {status_code}`	You will get the status code with error msg.	Kindly reach the developer with the exact msg.
	405	`HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.`	Number of request has exceeded for the cortex api	Kindly reach the developer with the exact msg.
	406	`HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.`	Number of request has exceeded for the cortex api	Kindly reach the developer with the exact msg.
	407	`HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.`	Number of request has exceeded for the cortex api	Kindly reach the developer with the exact msg.
	408	`HTTP Error occurred while retrieving events from cortex server, summary: {str(e)}, details: {details_str}`	You will get the status code with error msg	Kindly reach the developer with the exact msg.
	409	`Some error occurred while retrieving events from cortex server. Error details {str(e)}`	You will get the status code with error msg	Kindly reach the developer with the exact msg.

Change log

Release

Released on

Release type

Recommendations

v2.0.4

17 Jan 2025

Status

colour	Yellow
title	bug fixing

Recommended version

Expand

title	Details

Bug Fix

Fixed the duplication issue in the incidents service.

v2.0.3

04 Dec 2024

Status

colour	Blue
title	IMPROVEMENT

Upgrade

Expand

title	Details

Improvements

Added 2 new puller and a base puller
Improved unit tests to added some extra tests
Added checks for validating the start_time and if the time is not in future

v2.0.2

07 Nov 2024

Status

colour	Blue
title	IMPROVEMENT

Status

colour	Yellow
title	bug fixing

Upgrade

Expand

title	Details

Improvements

Updated DCSDK from 1.12.4 to 1.13.1
- Fixed bug related to module_global_status value in message_metrics
- PEP8 Cleanup
- New metric endpoint (http://0.0.0.0:3000/metrics)
- Changed some metric structure that were already sent to Devo before (devo.collector.metric.*)
- Improved MacOS compatibility (for the development phase)
- Updated DevoSDK to version 6.0.0
- Puller and PullerSetup now have the same id structure
- Changed some console log traces to DEBUG
- Fixed bug related to rate_limiter object (the object was not properly internally released)
- Improved Filesystem persistence behavior
- New libraries installed:
  - uvicorn==0.31.1
  - fastapi==0.115.0
  - Updated libraries:
  - jsonschema==4.19.1 -> jsonschema==4.23.0
  - psutil==5.9.6 -> psutil==5.9.8
  - python-dateutil==2.8.2 -> python-dateutil==2.9.0.post0

Bug

Modified paging logic so that the collector responds correctly to controlled stops.

v2.0.1

30 Sep 2024

Status

colour	Yellow
title	bug fixing

Status

colour	Blue
title	IMPROVEMENT

Recommended versionUpgrade

Expand

title	Details

Improvements

upgraded the docker base image to 1.3.1

Bug fixing

Added condition to resolve the infinite issue

v2.0.0

06 Sep 2024

Status

colour	Yellow
title	bug fixing

Status

colour	Blue
title	IMPROVEMENT

Upgrade

Expand

title	Details

Improvements

Added new services: violations, audit_management and all_alert.
Modified the table of incident alert.
Removed the usage of alert multi event table.
Using v2 API for alert service.
Unified all the puller services in a single puller.
Moved api_endpoint in the collector definition.

v1.4.0

23 Aug 2024

Status

colour	Blue
title	IMPROVEMENT

Upgrade

Expand

title	Details

Improvements:

Added start_time as an optional parameters for both of the services.
Added deduplication logic for both services
Updated Docker image base to version v1.3.0 in Dockerfile
Updated DCSDK from v1.11.1 to v1.12.4
- Added new sender for relay in house + TLS
- Added persistence functionality for gzip sending buffer
- Added Automatic activation of gzip sending
- Improved behaviour when persistence fails
- Upgraded DevoSDK dependency
- Fixed console log encoding
- Restructured python classes
- Improved behaviour with non-utf8 characters
- Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)
- New persistence format/structure (compression in some cases)
- Removed dmesg execution (It was invalid for docker execution)
- Applied changes to make DCSDK compatible with MacOS
- Improved behavior with non-utf8 characters
- Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)
- New persistence format/structure (compression in some cases)
- Removed dmesg execution (It was invalid for docker execution)
- DevoSDK has been updated to version 5.4.0

v1.3.0

23 May 2024

Status

colour	Blue
title	IMPROVEMENT

Upgrade

Expand

title	Details

Improvements:

Upgrade DC SDK to the latest version 1.11.1
Upgrade the Docker base image to 1.2.0

v1.2.0

20 Mar 2024

Status

colour	Blue
title	IMPROVEMENT

Initial version

Expand

title	Details

Improvements:

Added 'start_time' in config file for alerts service
Added logs

Version	Old Version 10	New Version Current
Changes made by	MD Tausif	MD Tausif
Saved on	Nov 27, 2024	Jan 17, 2025

Content Comparison

Versions Compared

Key

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Change log

Change log