Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Rw ui tabs macro
Rw tab
titleCloud collector

The Collector Server is a managed platform that allows running sets of different collectors grouped by Devo domain destinations.

To run an instance of this data collector, the next steps must be followed:

  1. In the Collector Server GUI, access the domain where you want to create this instance, click Add Collector, search for “Cortex XDR - Integrations Factory”, then click on the result.

  2. In the Version field, select the latest value.

  3. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).

  4. In the Parameters section, establish the Collector Parameters as follows below:Collector services detail

Info

Please, replace the placeholders <api_key_value>, <api_key_id_value>, and <api_fqdn_value> in the next section with the values obtained in previous sections of this document, except the <short_unique_identifier> that can have the value you choose. Do not substitute the occurrences of {api_fqdn}.

Code Block
{
  "global_overrides": {
    "debug": <debug_status>
  },
  "inputs": {
    "cortex_xdr": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "api_key": "<api_key>",
        "api_key_id": "<api_key_id>"
      },
      "services": {
        "incidents": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "include_incident_alerts": "<include_incident_alerts>",
          "override_devo_tag": "<override_devo_tag>"
          "override_incident_alert_tag": "<override_incident_alert_tag>"
        },
        "alerts": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "all_alerts": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "violations": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "audit_managements": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        }
      }
    }
  }
}
Note

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Info

Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

<short_unique_id>

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

Note

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

<input_status>

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

<api_key>

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

<api_key_id>

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id}

<api_fqdn>

str

Mandatory

Minimum Length 1

The FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

<request_period_in_seconds>

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

<override_devo_tag>

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

<override_incident_alert_tag>

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts( Extra incident endpoint). Ex: my.app.devo.Incident_alert

<include_incident_alerts>

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data (Extra incidents data

for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

<start_time>

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

Rw tab
titleOn-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block
<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml 
Note

Replace <product_name> with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

image-20240528-122729.png
Note

Replace <product_name> with the proper value.

Editing the config.yaml file

Code Block
globals:
  debug: false
  id: not_used
  name: cortex_xdr
  persistence:
    type: filesystem
    config:
      directory_name: state

outputs:
  devo_1:
    type: devo_platform
    config:
      address: collector-us.devo.io
      port: 443
      type: SSL
      chain: chain.crt
      cert: <devo_domain>.crt
      key: <devo_domain>.key
  console_1:
    type: console

inputs:
  cortex_xdr:
    id: <short_unique_id>
    enabled: true
    credentials:
      api_key: <api_key>
      api_key_id: <api_key_id>
    services:
      incidents:
        api_fqdn: <api_fqdn>
        request_period_in_seconds : <request_period_in_seconds> #optional
        start_time: <start_time> #optional
        include_incident_alerts: <include_incident_alerts> #Optional
        override_devo_tag: <override_devo_tag> #optional
        override_incident_alert_tag: <override_incident_alert_tag> #optional
      alerts:
        api_fqdn: <api_fqdn>
        start_time: <start_time> # Example 2024-01-01T01:50:00Z
        request_period_in_seconds: <request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      all_alerts:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      audit_managements:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      violations:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
Note
All defined service entities will be executed by the collector. If
Note

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Info

Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

short_unique_id

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

Note

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

api_key

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

api_key_id

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id} while calling the API.

api_fqdn

str

Mandatory

Minimum Length 1

The {api_fqdn} FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

request_period_in_seconds

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

override_devo_tag

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

override_incident_alert_tag

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts ( Extra incident endpoint). Ex: my.app.devo.Incident_alert

include_incident_alerts

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data (Extra incidents data for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

start_time

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-cortex_xdr_if-docker-image-2.0.13

1f7a3dd54371e26538b35fced4a8965bb7534ce7f84360065fc815711c856cb88e821a375fe7dfa275b5f29d343b157d42cd024e91f9e81d52e7816de339d6e5

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz | docker load
Note

Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block
docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>
Note

Replace <product_name>, <image_name> and <version> with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block
version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note

Replace <product_name>, <image_name> and <version> with the proper values.

...

Expand
titleTroubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.

The authentication was correct, but returned an unexpected status code

Error type

Error ID

Error message

Cause

Solution

SetupError

100

HTTP Error occurred while checking the server health for cortex

If 401 client error, credentials used are not valid.
or issue with the API request process.

Ensure correct credentials are used if 401 client error. Else Contact devo support team

101

Some error occurred while checking the server health for cortex. Error details

from remote source: {e}

You will get the status code with error msg.

Ensure that the collector has the necessary permissions and proper credentials to access the Cortex xdr API and contact the developer with exact error message

PullError

300

Error while fetching the event data for service {service_name} and error: {e}

You will get the status code with error msg.

This is an internal issue. Contact with Devo Support team.

PullError

300

Expected 200 status code, received {status code}

There has been an error upon API request process.

This is an internal issue. Contact with Devo Support team.

301

Expected 200 status code, received {status code}

You will get the status code with error msg.

Kindly reach the develop with the exact msg.

Change log

...

Release

...

Released on

...

Release type

...

ApiError

400

HTTP ERROR 400: Bad request: The server could not understand the request

Bad Request

Kindly reach out to the developer

401

HTTP ERROR 401: Unauthorized: Authentication is required and has failed or has not been provided

Credentials provided is not correct

Please check and provide the correct credentials

402

Unauthorized access. User does not have the required license type to run this API

User doesn't have the required permission

Kindly get required permission and reach the developer with the exact msg.

403

HTTP ERROR 500: Server Error: An error occurred on the server

Internal server error. An unverified status of API communication type error

Kindly reach the developer with the exact msg.

404

Request Error: Received status code {status_code}

You will get the status code with error msg.

Kindly reach the developer with the exact msg.

405

HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.

Number of request has exceeded for the cortex api

Kindly reach the developer with the exact msg.

406

HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.

Number of request has exceeded for the cortex api

Kindly reach the developer with the exact msg.

407

HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.

Number of request has exceeded for the cortex api

Kindly reach the developer with the exact msg.

408

HTTP Error occurred while retrieving events from cortex server, summary: {str(e)}, details: {details_str}

You will get the status code with error msg

Kindly reach the developer with the exact msg.

409

Some error occurred while retrieving events from cortex server. Error details {str(e)}

You will get the status code with error msg

Kindly reach the developer with the exact msg.

Change log

Release

Released on

Release type

Recommendations

v2.0.3

Status
colourBlue
titleIMPROVEMENT

Recommended version

Expand
titleDetails

Improvements

  • Added 2 new puller to extract from the base puller

  • Improved unit tests to added some extra tests

  • Added checks for validating the start_time and if the time is not in future

v2.0.2

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements

  • Updated DCSDK from 1.12.4 to 1.13.1

    • Fixed bug related to module_global_status value in message_metrics

    • PEP8 Cleanup

    • New metric endpoint (http://0.0.0.0:3000/metrics)

    • Changed some metric structure that were already sent to Devo before (devo.collector.metric.*)

    • Improved MacOS compatibility (for the development phase)

    • Updated DevoSDK to version 6.0.0

    • Puller and PullerSetup now have the same id structure

    • Changed some console log traces to DEBUG

    • Fixed bug related to rate_limiter object (the object was not properly internally released)

    • Improved Filesystem persistence behavior

    • New libraries installed:

      • uvicorn==0.31.1

      • fastapi==0.115.0

      • Updated libraries:

      • jsonschema==4.19.1 -> jsonschema==4.23.0

      • psutil==5.9.6 -> psutil==5.9.8

      • python-dateutil==2.8.2 -> python-dateutil==2.9.0.post0

Bug

  • Modified paging logic so that the collector responds correctly to controlled stops.

v2.0.1

Status
colourYellow
titlebug fixing

Status
colourBlue
titleIMPROVEMENT

Recommended versionUpgrade

Expand
titleDetails

Improvements

  • upgraded the docker base image to 1.3.1

Bug fixing

  • Added condition to resolve the infinite issue

v2.0.0

Status
colourYellow
titlebug fixing

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements

  • Added new services: violations, audit_management and all_alert.

  • Modified the table of incident alert.

  • Removed the usage of alert multi event table.

  • Using v2 API for alert service.

  • Unified all the puller services in a single puller.

  • Moved api_endpoint in the collector definition.

v1.4.0

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements:

  • Added start_time as an optional parameters for both of the services.

  • Added deduplication logic for both services

  • Updated Docker image base to version v1.3.0 in Dockerfile

  • Updated DCSDK from v1.11.1 to v1.12.4

    • Added new sender for relay in house + TLS

    • Added persistence functionality for gzip sending buffer

    • Added Automatic activation of gzip sending

    • Improved behaviour when persistence fails

    • Upgraded DevoSDK dependency

    • Fixed console log encoding

    • Restructured python classes

    • Improved behaviour with non-utf8 characters

    • Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • Applied changes to make DCSDK compatible with MacOS

    • Improved behavior with non-utf8 characters

    • Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • DevoSDK has been updated to version 5.4.0

v1.3.0

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements:

  • Upgrade DC SDK to the latest version 1.11.1

  • Upgrade the Docker base image to 1.2.0

v1.2.0

Status
colourBlue
titleIMPROVEMENT

 Initial version

Expand
titleDetails

Improvements:

  • Added 'start_time' in config file for alerts service

  • Added logs