Versions Compared

Version Old Version 9 New Version Current
Changes made by

Borja Moro Moreno

MD Tausif

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Rw ui tabs macro
Rw tab
titleCloud collector

The Collector Server is a managed platform that allows running sets of different collectors grouped by Devo domain destinations.

To run an instance of this data collector, the next steps must be followed:

  1. In the Collector Server GUI, access the domain where you want to create this instance, click Add Collector, search for “Cortex XDR - Integrations Factory”, then click on the result.

  2. In the Version field, select the latest value.

  3. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).

  4. In the Parameters section, establish the Collector Parameters as follows below:Collector services detail

Info

Please, replace the placeholders <api_key_value>, <api_key_id_value>, and <api_fqdn_value> in the next section with the values obtained in previous sections of this document, except the <short_unique_identifier> that can have the value you choose. Do not substitute the occurrences of {api_fqdn}.

Code Block
{
  "global_overrides": {
    "debug": <debug_status>
  },
  "inputs": {
    "cortex_xdr": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "api_key": "<api_key>",
        "api_key_id": "<api_key_id>"
      },
      "services": {
        "incidents": {
          "api_fqdn": "<api_fqdn>",
          },"request_period_in_seconds" : <request_period_in_seconds>",
          "alertsstart_time": {"<start_time>",
          "apiinclude_incident_fqdnalerts": "<api<include_incident_fqdn>"
     alerts>",
  },         "alloverride_devo_alertstag": {"<override_devo_tag>"
          "api_fqdnoverride_incident_alert_tag": "<api_fqdn><override_incident_alert_tag>"
        },
        "violationsalerts": {
          "api_fqdn": "<api_fqdn>",
        },  "request_period_in_seconds" : <request_period_in_seconds>",
          "auditstart_managementstime": {"<start_time>",
          "apioverride_devo_fqdntag": "<api<override_devo_fqdn>tag>"
        },
        "all_alerts": {
 }     }   }
}
Note
All 
 "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "audit_managements": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        }
      }
    }
  }
}
Note

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Info

Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

<short_unique_id>

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

Note

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

<input_status>

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

<api_key>

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

<api_key_id>

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id}

<api_fqdn>

str

Mandatory

Minimum Length 1

The FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

<request_period_in_seconds>

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

<override_devo_tag>

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

<override_incident_alert_tag>

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts( Extra incident endpoint). Ex: my.app.devo.Incident_alert

<include_incident_alerts>

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘alse’ we will not be able to get incident alerts data (Extra incidents data). Ex : true, false.devo.Incident_alert

<include_incident_alerts>

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data

(Extra incidents data for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

Info

Note: For This endpoint, the data gets ingested in the table edr.cortex_xdr.incident_alert and this table may contains duplicates because we are using incident_id to get the data and these incident_ids can be duplicates.
We are using modification_time instead of creation_time to filter the incident_ids causing duplicates.

<start_time>

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

Rw tab
titleOn-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block
<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml
Note

Replace <product_name> with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

image-20240528-122729.png
Note

Replace <product_name> with the proper value.

Editing the config.yaml file

Code Block
globals:
  debug: false
  id: not_used
  name: cortex_xdr
  persistence:
    type: filesystem
    config:
      directory_name: state

outputs:
  devo_1:
    type: devo_platform
    config:
      address: collector-us.devo.io
      port: 443
      type: SSL
      chain: chain.crt
      cert: <devo_domain>.crt
      key: <devo_domain>.key
  console_1:
    type: console

inputs:
  cortex_xdr:
    id: <short_unique_id>
    enabled: true
    credentials:
      api_key: <api_key>
      api_key_id: <api_key_id>
    services:
      incidents:
        api_fqdn: <api_fqdn>
        request_period_in_seconds : <request_period_in_seconds> #optional
        start_time: <start_time> #optional
        include_incident_alerts: <include_incident_alerts> #Optional
        override_devo_tag: <override_devo_tag> #optional
        override_incident_alert_tag: <override_incident_alert_tag> #optional
      alerts:
        api_fqdn: <api_fqdn>
        start_time: <start_time> # Example 2024-01-01T01:50:00Z_incident_alerts: <include_incident_alerts> #Optional
          requestoverride_perioddevo_in_secondstag: <request<override_perioddevo_in_seconds>tag> #optional
        override_devoincident_alert_tag: <override_incident_devoalert_tag> #optional
      all_alerts:
        api_fqdn: <api_fqdn>
        start_time: <opt<start_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request<request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      auditall_managementsalerts:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      violationsaudit_managements:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
Note

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Info

Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

short_unique_id

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

Note

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

api_key

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

api_key_id

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id} while calling the API.

api_fqdn

str

Mandatory

Minimum Length 1

The {api_fqdn} FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

request_period_in_seconds

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

override_devo_tag

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

override_incident_alert_tag

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts ( Extra incident endpoint). Ex: my.app.devo.Incident_alert

include_incident_alerts

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data (Extra incidents data). Ex : true, falsefalse’ we will not be able to get incident alerts data (Extra incidents data for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

Info

Note: For This endpoint, the data gets ingested in the table edr.cortex_xdr.incident_alert and this table may contains duplicates because we are using incident_id to get the data and these incident_ids can be duplicates.
We are using modification_time instead of creation_time to filter the incident_ids causing duplicates.

start_time

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-cortex_xdr_if-docker-image-2.0.14

1f7a3dd54371e26538b35fced4a8965bb7534ce7f84360065fc815711c856cb86a1c62cfeb366873d2fcc67dbb2ccefc69841941e99b054f1ab8b27aca8752f5

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz | docker load
Note

Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block
docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>
Note

Replace <product_name>, <image_name> and <version> with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block
version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note

Replace <product_name>, <image_name> and <version> with the proper values.

...

2024-09-06T15:07:25.139 INFO MainProcess::MainThread -> Started all objects from "MainProcess" process

logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module, including all services together:

Expand
titleInitialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module, including all services together:

Code Block
Code Block
2024-09-06T15:07:25.139    INFO MainProcess::MainThread -> Started all objects from "MainProcess" process
2024-09-06T15:07:25.144    INFO InputProcess::MainThread -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork")
2024-09-06T15:07:25.220    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gz
2024-09-06T15:07:25.223    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.144224    INFO InputProcessOutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork") [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec3289"
2024-09-06T15:07:25.220227    INFO OutputProcessInputProcess::MainThread -> DevoSenderManagerCortexXdrBasePuller(standardcortex_senders,manager,devo_1xdr#1243598,incidents#predefined) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gzused/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.223231    INFO OutputProcessInputProcess::MainThread -> DevoSenderManagerCortexXdrBasePuller(standardcortex_senders,manager,devo_1xdr#1243598,incidents#predefined) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.224231    INFO OutputProcessInputProcess::MainThread -> DevoSenderManagerCortexXdrBasePuller(standardcortex_senders,manager,devo_1xdr#1243598,incidents#predefined) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 12), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec32896a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.227232    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.232    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.jsonDevoSender;standard_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.231233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] ThereChecking is no data persisted with the latest format, any previous if old persisted data willmust be migratedremoved
2024-09-06T15:07:25.231233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.232233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.232235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] CreatedNo previous persistence instance file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;standard_senders;devo_sender_0.json.gzb1ee30ea2c4b3be1eb50c8b2ca80d8d8"
2024-09-06T15:07:25.233235    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
Starting the execution of init_variables()
2024-09-06T15:07:25.233236    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"MainThread -> Validating service metadata
2024-09-06T15:07:25.233238    INFO InputProcessOutputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefinedOutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] NoCreated previous persistence file exists (Version 1)instance, filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"not_used/OutputStandardConsumer;standard_senders;0.json.gz
2024-09-06T15:07:25.235241    INFO OutputProcess::MainThread -> DevoSenderOutputStandardConsumer(standard_senders,devo_senderconsumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.235241    INFO OutputProcess::MainThread -> DevoSenderOutputStandardConsumer(standard_senders,devo_senderconsumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/b1ee30ea2c4b3be1eb50c8b2ca80d8d8de28663278a264d05d531fbc1db51a93"
2024-09-06T15:07:25.235242    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Starting the execution of init_variables()Validating defined module definition
2024-09-06T15:07:25.236243    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.238    INFO OutputProcess::MainThread -> OutputStandardConsumerDevoSenderManager(standardlookup_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputStandardConsumerDevoSenderManager;standardlookup_senders;0devo_1.json.gz
2024-09-06T15:07:25.241246    INFO OutputProcess::MainThread -> OutputStandardConsumerDevoSenderManager(standardlookup_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.241246    INFO OutputProcess::MainThread -> OutputStandardConsumerDevoSenderManager(standardlookup_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/de28663278a264d05d531fbc1db51a93a28c135f3fc69ece87d8fc2fdb029e8c"
2024-09-06T15:07:25.242251    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.243    INFO OutputProcess::MainThread -> DevoSenderManagerDevoSender(lookup_senders,manager,devo_sender_10) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManagerDevoSender;lookup_senders;devo_1sender_0.json.gz
2024-09-06T15:07:25.246252    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSenderManagerDevoSender(lookup_senders,manager,devo_1sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.246253    INFO OutputProcess::MainThread -> DevoSenderManagerDevoSender(lookup_senders,manager,devo_sender_10) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/a28c135f3fc69ece87d8fc2fdb029e8cdf8895fef2a509cbd87fcc9850dc0c81"
2024-09-06T15:07:25.251255    INFO OutputProcess::MainThread -> DevoSenderOutputLookupConsumer(lookup_senders,devo_senderconsumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderOutputLookupConsumer;lookup_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.252256    INFO InputProcess::MainThread -> Validating commonservice input config
2024-09-06T15:07:25.253257    INFO OutputProcess::MainThread -> DevoSenderOutputLookupConsumer(lookup_senders,devo_senderconsumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.253257    INFO OutputProcess::MainThread -> DevoSenderOutputLookupConsumer(lookup_senders,devo_senderconsumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/df8895fef2a509cbd87fcc9850dc0c81865a79c1b99ad39b22becc235c9732cb"
2024-09-06T15:07:25.255259    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.259    INFO OutputProcess::MainThread -> OutputLookupConsumerDevoSenderManager(lookupinternal_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputLookupConsumerDevoSenderManager;lookup_senders;0.json.gzinternal_senders;devo_1.json.gz
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.256261    INFO InputProcess::MainThread -> Validating service input configAdding raw config to the collector store
2024-09-06T15:07:25.257261    INFO OutputProcess::MainThread -> OutputLookupConsumerDevoSenderManager(lookupinternal_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.257261    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> OutputLookupConsumerDevoSenderManager(lookupinternal_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/865a79c1b99ad39b22becc235c9732cbe784c94a6fe28de5937e50674407d82e"
2024-09-06T15:07:25.259265    INFO InputProcess::MainThread -> Running overriding rulesCortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.259265    INFO OutputProcess::MainThread -> DevoSenderManagerDevoSender(internal_senders,manager,devo_sender_10) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManagerDevoSender;internal_senders;devo_1.json.gz
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
sender_0.json.gz
2024-09-06T15:07:25.260268    INFO InputProcessOutputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Adding raw config to the collector store
DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.261268    INFO OutputProcess::MainThread -> DevoSenderManagerDevoSender(internal_senders,manager,devo_sender_10) -> [EMERGENCY_PERSISTENCE_SYSTEM] ThereNo isprevious nopersistence datafile persistedexists withto themigrate latest(Version format1), any previous persisted data will be migrated
filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-09-06T15:07:25.261269    INFO InputProcess::MainThread -> Running custom validation rules CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.261271    INFO OutputProcess::MainThread -> DevoSenderManagerOutputInternalConsumer(internal_senders,manager,devo_1_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] NoCreated previous persistence file exists to migrate (Version 1)instance, filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/e784c94a6fe28de5937e50674407d82e"/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-09-06T15:07:25.265272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables()(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.265272    INFO OutputProcessInputProcess::MainThread -> DevoSenderCortexXdrBasePuller(internalcortex_senders,devo_sender_0xdr#1243598,alerts#predefined) -> [EMERGENCY_PERSISTENCE_SYSTEM] CreatedNo previous persistence instance file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.268272    INFO OutputProcess::MainThread -> DevoSenderOutputInternalConsumer(internal_senders,devo_senderconsumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.268272    INFO OutputProcessInputProcess::MainThread -> DevoSenderCortexXdrBasePuller(internalcortex_senders,devo_sender_0xdr#1243598,alerts#predefined) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.269273    INFO InputProcessOutputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefinedOutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] CreatedNo previous persistence instance file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.jsonxdr/state/10dd360c86621afd5a28a029a0dddcf6"
2024-09-06T15:07:25.271273    INFO OutputProcess::MainThread -> OutputInternalConsumerDevoSender(internalstandard_senders,devo_consumersender_0) -> [EMERGENCY_Starting thread
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz Checking if old persisted data must be removed
2024-09-06T15:07:25.272274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] ThereNo isprevious nopersistence datafile persistedexists with the latest format, any previous persisted data will be migrated
(Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.272274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 21), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.272274    INFO OutputProcess::MainThread -> OutputInternalConsumerDevoSenderManagerMonitor(internalstandard_senders,devo_consumer_01) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
Starting thread (every 300 seconds)
2024-09-06T15:07:25.275    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.272275    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.275    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.273276    INFO OutputProcess::MainThread -> OutputInternalConsumer(internalOutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] NoRecovering previousany persistenceavailable filecontent existsfrom tothe migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/10dd360c86621afd5a28a029a0dddcf6"
persistence system
2024-09-06T15:07:25.273276    INFO OutputProcessInputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
Validating service metadata
2024-09-06T15:07:25.274276    INFO InputProcessOutputProcess::MainThread -> CortexXdrBasePullerDevoSender(cortexlookup_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.274277    INFO InputProcessOutputProcess::MainThread -> CortexXdrBasePullerDevoSenderManagerMonitor(cortexlookup_xdr#1243598senders,alerts#predefineddevo_1) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"Starting thread (every 300 seconds)
2024-09-06T15:07:25.274279    INFO InputProcessOutputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefinedDevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] NoNothing previousavailable persistencein filethe exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
persistence system
2024-09-06T15:07:25.274279    INFO OutputProcess::MainThread -> DevoSenderManagerMonitorDevoSenderManager(standard_senders,manager,devo_1) -> Starting thread (every 300 seconds)[EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.275279    INFO OutputProcess::MainThread -> DevoSenderManager(standardlookup_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.275279    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables()
Validating defined module definition
2024-09-06T15:07:25.275280    INFO OutputProcess::DevoSenderManager(standardlookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276280    INFO OutputProcess::OutputStandardConsumerOutputLookupConsumer(standardlookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276280    INFO InputProcessOutputProcess::MainThread -> Validating service metadata DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.276282    INFO OutputProcess::MainThread -> DevoSenderDevoSenderManagerMonitor(lookupinternal_senders,devo_sender_01) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.277283    INFO OutputProcess::MainThreadOutputStandardConsumer(standard_senders_consumer_0) -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds) [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.279283    INFO OutputProcess::DevoSenderManagerOutputLookupConsumer(standardlookup_senders,manager,devo_1_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.279283    INFO OutputProcess::DevoSenderManagerOutputLookupConsumer(standardlookup_senders,manager,devo_1_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.279285    INFO OutputProcess::MainThread -> DevoSenderManager(lookupinternal_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.279    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.280285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] RecoveringNothing any available contentin from the persistence system
2024-09-06T15:07:25.280285    INFO OutputProcess::OutputLookupConsumerDevoSenderManager(lookup_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting threadElapsed seconds: 0.01
2024-09-06T15:07:25.282285    INFO OutputProcess::MainThread -> DevoSenderManagerMonitorDevoSenderManager(internal_senders,manager,devo_1) -> Starting thread (every 300 seconds)[EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.283286    INFO OutputProcess::OutputStandardConsumerOutputInternalConsumer(standardinternal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering Nothingany available content infrom the persistence system
2024-09-06T15:07:25.283287    INFO OutputProcessInputProcess::OutputStandardConsumer(standard_senders_consumer_0)MainThread -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01Validating common input config
2024-09-06T15:07:25.283289    INFO OutputProcess::OutputLookupConsumerDevoSenderManager(lookupinternal_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283289    INFO OutputProcess::OutputLookupConsumerDevoSenderManager(lookupinternal_senders_consumer_0,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.285294    INFO OutputProcessInputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting threadValidating service input config
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system298    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.285299    INFO OutputProcessInputProcess::DevoSenderManager(lookup_senders,manager,devo_1)MainThread -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 Validating the rate limiter config given by the user
2024-09-06T15:07:25.285300    INFO OutputProcessInputProcess::DevoSenderManager(internal_senders,manager,devo_1)MainThread -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system<requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.286301    INFO OutputProcessInputProcess::OutputInternalConsumer(internal_senders_consumer_0)MainThread -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content fromAdding raw config to the persistencecollector systemstore
2024-09-06T15:07:25.287302    INFO InputProcess::MainThread -> ValidatingRunning commoncustom inputvalidation configrules
2024-09-06T15:07:25.289302    INFO OutputProcessInputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables()
Expand
titleEvents delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Code Block
2024-09-06T15:0732:2547.289810    INFO OutputProcess::DevoSenderManagerDevoSenderManagerMonitor(internallookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds Number of available senders: 1, sender manager internal queue size: 0.00
2024-09-06T15:0732:2547.294810    INFO InputProcessOutputProcess::MainThreadDevoSenderManagerMonitor(lookup_senders,devo_1) -> Validating service input config
enqueued_elapsed_times_in_seconds_stats: {}
2024-09-06T15:0732:2547.298810    INFO InputProcessOutputProcess::MainThreadDevoSenderManagerMonitor(lookup_senders,devo_1) -> Running overriding rules Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2024-09-06T15:0732:2547.299811    INFO InputProcessOutputProcess::MainThreadDevoSenderManagerMonitor(lookup_senders,devo_1) -> Validating the rate limiter config given by the user
Lookup - Total number of messages sent: 0, messages sent since "2024-09-06T1506 09:0757:25.300    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.301    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables()
Expand
titleEvents delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Code Block
2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Number of available senders: 1, sender manager internal queue size: 0
2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2024-09-06T15:32:47.81147.810025+00:00": 0 (elapsed 0.000 seconds)

By default, these information traces will be displayed every 10 minutes.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 0, messages sent since "2024-09-06 09:57:47.810025+00:00": 0 (elapsed 0.000 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

  • 0 events were sent to Devo since the collector started.

  • The last checkpoint timestamp was 2024-09-06 09:57:47.810025+00:00.

  • 0 events where sent to Devo between the last UTC checkpoint and now.

  • Those 0 events required 0.000 seconds to be delivered.

By default these traces will be shown every 10 minutes.

...

Error type

...

Error ID

...

Error message

...

Cause

...

Solution

...

SetupError

...

100

...

HTTP Error occurred while checking the server health for cortex

...

If 401 client error, credentials used are not valid.
or issue with the API request process.

...

Ensure correct credentials are used if 401 client error. Else Contact devo support team

...

101

...

Some error occurred while checking the server health for cortex. Error details

...

The authentication was correct, but returned an unexpected status code.

...

This is an internal issue. Contact with Devo Support team.

...

PullError

...

300

...

Expected 200 status code, received {status code}

...

There has been an error upon API request process.

...

This is an internal issue. Contact with Devo Support team.

...

301

...

Expected 200 status code, received {status code}

...

You will get the status code with error msg.

...

Kindly reach the develop with the exact msg.

Change log

...

Release

...

Released on

...

Release type

...

Expand
titleCheck memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

Info

Some collectors can be tuned to reduce the memory impact when pulling data in large infrastructures.

  • The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

  • The global pressure of the available memory is displayed in the global value.

  • Al metrics (Global, RSS, VMS) include the value before freeing and after: previous -> after freeing memory

Code Block
2024-09-06T15:07:25.494    INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB)
2024-09-06T15:09:25.375    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1)MainThread -> Lookup - Total number of messages sent: 0, messages sent since "2024-09-06 09:57:47.810025+00:00": 0 (elapsed 0.000 seconds)

By default, these information traces will be displayed every 10 minutes.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 0, messages sent since "2024-09-06 09:57:47.810025+00:00": 0 (elapsed 0.000 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

  • 0 events were sent to Devo since the collector started.

  • The last checkpoint timestamp was 2024-09-06 09:57:47.810025+00:00.

  • 0 events where sent to Devo between the last UTC checkpoint and now.

  • Those 0 events required 0.000 seconds to be delivered.

By default these traces will be shown every 10 minutes.

Expand
titleCheck memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

Info

Some collectors can be tuned to reduce the memory impact when pulling data in large infrastructures.

  • The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

  • The global pressure of the available memory is displayed in the global value.

  • Al metrics (Global, RSS, VMS) include the value before freeing and after: previous -> after freeing memory

Code Block
2024-09-06T15:07:25.494    INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB)
2024-09-06T15:09:25.375    INFO OutputProcess::MainThread -> [GC] global: 42.4% -> 42.4%, process: RSS(44.46MiB -> 44.46MiB), VMS(929.14MiB -> 929.14MiB)

Differences between RSS and VMS memory usage:

  • RSS is the Resident Set Size, which is the actual physical memory the process is using

  • VMS is the Virtual Memory Size which is the virtual memory that process is using

Expand
titleEnable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

  • To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.

  • To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

...

titleTroubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.

[GC] global: 42.4% -> 42.4%, process: RSS(44.46MiB -> 44.46MiB), VMS(929.14MiB -> 929.14MiB)

Differences between RSS and VMS memory usage:

  • RSS is the Resident Set Size, which is the actual physical memory the process is using

  • VMS is the Virtual Memory Size which is the virtual memory that process is using

Expand
titleEnable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

  • To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.

  • To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Expand
titleTroubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.

Error type

Error ID

Error message

Cause

Solution

SetupError

100

HTTP Error occurred while checking the server health for cortex from remote source: {e}

You will get the status code with error msg.

Ensure that the collector has the necessary permissions and proper credentials to access the Cortex xdr API and contact the developer with exact error message

PullError

300

Error while fetching the event data for service {service_name} and error: {e}

You will get the status code with error msg.

This is an internal issue. Contact with Devo Support team.

ApiError

400

HTTP ERROR 400: Bad request: The server could not understand the request

Bad Request

Kindly reach out to the developer

401

HTTP ERROR 401: Unauthorized: Authentication is required and has failed or has not been provided

Credentials provided is not correct

Please check and provide the correct credentials

402

Unauthorized access. User does not have the required license type to run this API

User doesn't have the required permission

Kindly get required permission and reach the developer with the exact msg.

403

HTTP ERROR 500: Server Error: An error occurred on the server

Internal server error. An unverified status of API communication type error

Kindly reach the developer with the exact msg.

404

Request Error: Received status code {status_code}

You will get the status code with error msg.

Kindly reach the developer with the exact msg.

405

HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.

Number of request has exceeded for the cortex api

Kindly reach the developer with the exact msg.

406

HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.

Number of request has exceeded for the cortex api

Kindly reach the developer with the exact msg.

407

HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.

Number of request has exceeded for the cortex api

Kindly reach the developer with the exact msg.

408

HTTP Error occurred while retrieving events from cortex server, summary: {str(e)}, details: {details_str}

You will get the status code with error msg

Kindly reach the developer with the exact msg.

409

Some error occurred while retrieving events from cortex server. Error details {str(e)}

You will get the status code with error msg

Kindly reach the developer with the exact msg.

Change log

Release

Released on

Release type

Recommendations

v2.0.4

Status
colourYellow
titlebug fixing

Recommended version

Expand
titleDetails

Bug Fix

  • Fixed the duplication issue in the incidents service.

v2.0.3

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements

  • Added 2 new puller and a base puller

  • Improved unit tests to added some extra tests

  • Added checks for validating the start_time and if the time is not in future

v2.0.2

Status
colourBlue
titleIMPROVEMENT

Status
colourYellow
titlebug fixing

Upgrade

Expand
titleDetails

Improvements

  • Updated DCSDK from 1.12.4 to 1.13.1

    • Fixed bug related to module_global_status value in message_metrics

    • PEP8 Cleanup

    • New metric endpoint (http://0.0.0.0:3000/metrics)

    • Changed some metric structure that were already sent to Devo before (devo.collector.metric.*)

    • Improved MacOS compatibility (for the development phase)

    • Updated DevoSDK to version 6.0.0

    • Puller and PullerSetup now have the same id structure

    • Changed some console log traces to DEBUG

    • Fixed bug related to rate_limiter object (the object was not properly internally released)

    • Improved Filesystem persistence behavior

    • New libraries installed:

      • uvicorn==0.31.1

      • fastapi==0.115.0

      • Updated libraries:

      • jsonschema==4.19.1 -> jsonschema==4.23.0

      • psutil==5.9.6 -> psutil==5.9.8

      • python-dateutil==2.8.2 -> python-dateutil==2.9.0.post0

Bug

  • Modified paging logic so that the collector responds correctly to controlled stops.

v2.0.1

Status
colourYellow
titlebug fixing

Status
colourBlue
titleIMPROVEMENT

Recommended versionUpgrade

Expand
titleDetails

Improvements

  • upgraded the docker base image to 1.3.1

Bug fixing

  • Added condition to resolve the infinite issue

v2.0.0

Status
colourYellow
titlebug fixing

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements

  • Added new services: violations, audit_management and all_alert.

  • Modified the table of incident alert.

  • Removed the usage of alert multi event table.

  • Using v2 API for alert service.

  • Unified all the puller services in a single puller.

  • Moved api_endpoint in the collector definition.

v1.4.0

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements:

  • Added start_time as an optional parameters for both of the services.

  • Added deduplication logic for both services

  • Updated Docker image base to version v1.3.0 in Dockerfile

  • Updated DCSDK from v1.11.1 to v1.12.4

    • Added new sender for relay in house + TLS

    • Added persistence functionality for gzip sending buffer

    • Added Automatic activation of gzip sending

    • Improved behaviour when persistence fails

    • Upgraded DevoSDK dependency

    • Fixed console log encoding

    • Restructured python classes

    • Improved behaviour with non-utf8 characters

    • Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • Applied changes to make DCSDK compatible with MacOS

    • Improved behavior with non-utf8 characters

    • Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • DevoSDK has been updated to version 5.4.0

v1.3.0

Status
colourBlue
titleIMPROVEMENT

Upgrade

Expand
titleDetails

Improvements:

  • Upgrade DC SDK to the latest version 1.11.1

  • Upgrade the Docker base image to 1.2.0

v1.2.0

Status
colourBlue
titleIMPROVEMENT

 Initial version

Expand
titleDetails

Improvements:

  • Added 'start_time' in config file for alerts service

  • Added logs