View Source

Overview

Cortex XDR delivers enterprise-wide protection by analyzing data from any source to stop sophisticated attacks.

Eliminate blind spots with complete visibility
Simplify security operations to cut mean time to respond (MTTR)
Harness the scale of the cloud for AI and analytics
Lower costs by consolidating tools and improving SOC efficiency.

Cortex XDR collector migration guide (from 1.x.x to 2.x.x)

If you need to migrate an old collector version to a more recent one, please check the migration process in this article.

Devo collector features

Feature	Details
Allow parallel downloading (`multipod`)	`not allowed`
Running environments	`collector server` `on-premise`
Populated Devo events	`table`
Flattening preprocessing	`no`
Requires IP Whitelisting	`no`

Data sources

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Alerts

Get a list of alerts with multiple events.

https://<api-yourfqdn>/public_api/v2/alerts/get_alerts_multi_events

alerts

edr.cortex_xdr.alerts_multi

You can override this in override_devo_tag in the incident module definition.

v2.0.0

Incidents

Get Incidents:
Get a list of incidents filtered by a list of incident IDs, modification time, or creation time.

Get Incident Alerts:
Get extra data fields of a specific incident including alerts and key artifacts.

List Incidents:
https://<api-yourfqdn>/public_api/v1/incidents/get_incidents

List Incident Alerts:

https://<api-yourfqdn>/public_api/v1/incidents/get_incident_extra_data

incidents

List Incidents:

edr.cortex_xdr.incidents

You can override this in override_devo_tag in the incident module definition.

List Incident Alerts:

edr.cortex_xdr.incident_alert

You can override this in override_devo_tag in the incident module definition.

v2.0.0

All Alerts

Get a list of alerts.

https://<api-yourfqdn>/public_api/v1/alerts/get_alerts

all_alerts

edr.cortex_xdr.all_alert

You can override this in override_devo_tag in the incident module definition.

v2.0.0

Audit Managements

Get audit management logs.

https://<api-yourfqdn>/public_api/v1/audits/management_logs

audit_management

edr.cortex_xdr.audit_management

You can override this in override_devo_tag in the incident module definition.

v2.0.0

Vendor configuration

Generate API key

Log in to your Cortex XDR console with an administrative account.
Go to the Settings menu (gear icon).
Go to API Keys under the Integrations section.
Click Add Key to generate a new API key.
Select the roles or permissions you want to assign to this key (these determine the actions the key can perform).
Click Create for generating the API Key along with an API Key ID. Save these securely, as you may not be able to retrieve them later.

API FQDN

The API FQDN (Fully Qualified Domain Name) can be found in the documentation or directly in the URL when you are logged into your Cortex XDR console.
It will usually be in the format: https://<YOUR_REGION>.xdr.paloaltonetworks.com

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details
`api_key`	The API Key is your unique identifier for Cortex XDR.
`api_key_id`	The API Key ID is your unique token used to authenticate the API key for Cortex XDR.
`api_fqdn`	API FQDN is a unique host and domain name associated with each tenant for Cortex XDR.

See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Accepted authentication methods

Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials configuration block.

Authentication method	API key	API key ID	API FQDN
API KEY / API KEY ID / API FQDN

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

The Collector Server is a managed platform that allows running sets of different collectors grouped by Devo domain destinations.

To run an instance of this data collector, the next steps must be followed:

In the Collector Server GUI, access the domain where you want to create this instance, click Add Collector, search for “Cortex XDR - Integrations Factory”, then click on the result.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the Parameters section, establish the Collector Parameters as follows below:Collector services detail

Please, replace the placeholders <api_key_value>, <api_key_id_value>, and <api_fqdn_value> in the next section with the values obtained in previous sections of this document, except the <short_unique_identifier> that can have the value you choose. Do not substitute the occurrences of {api_fqdn}.

{
  "global_overrides": {
    "debug": <debug_status>
  },
  "inputs": {
    "cortex_xdr": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "api_key": "<api_key>",
        "api_key_id": "<api_key_id>"
      },
      "services": {
        "incidents": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "include_incident_alerts": "<include_incident_alerts>",
          "override_devo_tag": "<override_devo_tag>"
          "override_incident_alert_tag": "<override_incident_alert_tag>"
        },
        "alerts": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "all_alerts": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        },
        "audit_managements": {
          "api_fqdn": "<api_fqdn>",
          "request_period_in_seconds" : <request_period_in_seconds>",
          "start_time": "<start_time>",
          "override_devo_tag": "<override_devo_tag>"
        }
      }
    }
  }
}

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

<short_unique_id>

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

<input_status>

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

<api_key>

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

<api_key_id>

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id}

<api_fqdn>

str

Mandatory

Minimum Length 1

The FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

<request_period_in_seconds>

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

<override_devo_tag>

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

<override_incident_alert_tag>

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts( Extra incident endpoint). Ex: my.app.devo.Incident_alert

<include_incident_alerts>

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data (Extra incidents data

for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

<start_time>

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml

Replace <product_name> with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Replace <product_name> with the proper value.

Editing the config.yaml file

globals:
  debug: false
  id: not_used
  name: cortex_xdr
  persistence:
    type: filesystem
    config:
      directory_name: state

outputs:
  devo_1:
    type: devo_platform
    config:
      address: collector-us.devo.io
      port: 443
      type: SSL
      chain: chain.crt
      cert: <devo_domain>.crt
      key: <devo_domain>.key
  console_1:
    type: console

inputs:
  cortex_xdr:
    id: <short_unique_id>
    enabled: true
    credentials:
      api_key: <api_key>
      api_key_id: <api_key_id>
    services:
      incidents:
        api_fqdn: <api_fqdn>
        request_period_in_seconds : <request_period_in_seconds> #optional
        start_time: <start_time> #optional
        include_incident_alerts: <include_incident_alerts> #Optional
        override_devo_tag: <override_devo_tag> #optional
        override_incident_alert_tag: <override_incident_alert_tag> #optional
      alerts:
        api_fqdn: <api_fqdn>
        start_time: <start_time> # Example 2024-01-01T01:50:00Z
        request_period_in_seconds: <request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      all_alerts:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional
      audit_managements:
        api_fqdn: <api_fqdn>
        start_time: <opt_start_time> # Example 2024-01-01T01:50:00Z #optional
        request_period_in_seconds: <opt_request_period_in_seconds> #optional
        override_devo_tag: <override_devo_tag> #optional

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Please replace the placeholders with real world values following the description table below

Parameter

Data type

Type

Value range / Format

Details

short_unique_id

int

Mandatory

Minimum Lenght 5

Use this param to give a unique id to this input service.

This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

api_key

str

Mandatory

Minimum Length 1

The API Key is your unique identifier used as the Authorization:{key}.

api_key_id

str

Mandatory

Minimum Length 1

The API Key ID is your unique token used to authenticate the API Key. It is used in headers as x-xdr-auth-id:{key_id} while calling the API.

api_fqdn

str

Mandatory

Minimum Length 1

The {api_fqdn} FQDN is a unique host and domain name associated with each tenant. When you generate the API Key and Key ID, you are assigned an individual FQDN. ex: https://{api-fqdn}/public_api/v1/incidents/get_incidents

request_period_in_seconds

int

Optional

Minimum Length 1

Period in seconds used between each data pulling, this value will overwrite the default value 60 seconds

override_devo_tag

str

Optional

A devo tag

This parameter allows to define a custom devo tag.
ex: my.app.devo.service

override_incident_alert_tag

str

Optional

A Incident Alert tag

This Tag is only applicable for Incidents service to override the tag of Incident alerts ( Extra incident endpoint). Ex: my.app.devo.Incident_alert

include_incident_alerts

boolean

Optional

A boolean to Include Incidents alerts

By default the value of this boolean is ‘true’. If given ‘false’ we will not be able to get incident alerts data (Extra incidents data for endpoint: v1/incidents/get_incident_extra_data). Ex : true, false

start_time

str

Optional

start time in utc
format: %Y-%m-%dT%H:%M:%SZ

This parameter allows to get the data from provided start time. If not provided it will take current-time as time. Ex:- 2024-01-01T01:50:00Z

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-cortex_xdr_if-docker-image-2.0.3	`1f0498204db80d8c30d1528b9471f8a3d7767046`

Use the following command to add the Docker image to the system:

gunzip -c <image_file>-<version>.tgz | docker load

Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Replace <product_name>, <image_name> and <version> with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

IMAGE_VERSION=<version> docker-compose up -d

Replace <product_name>, <image_name> and <version> with the proper values.

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Verify data collection

Service components

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup is common for every service, so its output is displayed in this section.

On the other hand, Pullers are specific to each Service, so their outputs are shown in their corresponding sections below.

Setup output

A successful run has the following output messages for the setup module:

2024-09-06T15:07:25.139    INFO MainProcess::MainThread -> Started all objects from "MainProcess" process
2024-09-06T15:07:25.144    INFO InputProcess::MainThread -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork")
2024-09-06T15:07:25.220    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gz
2024-09-06T15:07:25.223    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.224    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec3289"
2024-09-06T15:07:25.227    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.232    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.232    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;standard_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/b1ee30ea2c4b3be1eb50c8b2ca80d8d8"
2024-09-06T15:07:25.235    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.236    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.238    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputStandardConsumer;standard_senders;0.json.gz
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/de28663278a264d05d531fbc1db51a93"
2024-09-06T15:07:25.242    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.243    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;lookup_senders;devo_1.json.gz
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/a28c135f3fc69ece87d8fc2fdb029e8c"
2024-09-06T15:07:25.251    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;lookup_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.252    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/df8895fef2a509cbd87fcc9850dc0c81"
2024-09-06T15:07:25.255    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz
2024-09-06T15:07:25.256    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/865a79c1b99ad39b22becc235c9732cb"
2024-09-06T15:07:25.259    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.259    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/e784c94a6fe28de5937e50674407d82e"
2024-09-06T15:07:25.265    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.265    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-09-06T15:07:25.269    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.271    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.272    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/10dd360c86621afd5a28a029a0dddcf6"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.274    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.275    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.275    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.275    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.276    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.277    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.279    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.279    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.280    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.282    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.285    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.286    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.287    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.294    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.298    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.299    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.300    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.301    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.307    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;all_alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.310    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.311    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf"
2024-09-06T15:07:25.311    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef"
2024-09-06T15:07:25.312    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.313    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf"
2024-09-06T15:07:25.313    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef"
2024-09-06T15:07:25.314    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.315    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.321    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.339    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.339    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.340    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.05
2024-09-06T15:07:25.341    INFO OutputProcess::MainThread -> [GC] global: 42.0% -> 42.0%, process: RSS(41.21MiB -> 41.21MiB), VMS(929.01MiB -> 929.01MiB)
2024-09-06T15:07:25.349    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.353    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.354    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.356    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.357    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.358    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.359    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.367    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;audit_managements;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.373    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.373    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2"
2024-09-06T15:07:25.373    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc"
2024-09-06T15:07:25.375    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.376    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2"
2024-09-06T15:07:25.376    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc"
2024-09-06T15:07:25.378    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.379    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.386    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.399    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.405    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.408    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.409    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.410    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.411    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.413    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.415    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.419    INFO InputProcess::MainThread -> InputThread(cortex_xdr,1243598) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.420    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,incidents,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.421    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Starting thread
2024-09-06T15:07:25.422    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) - Starting thread
2024-09-06T15:07:25.430 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.431    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,alerts,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.432    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,alerts#predefined) -> Starting thread
2024-09-06T15:07:25.433    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) - Starting thread
2024-09-06T15:07:25.435 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.438    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,all_alerts,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.440    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,all_alerts#predefined) -> Starting thread
2024-09-06T15:07:25.440    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) - Starting thread
2024-09-06T15:07:25.444 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.445    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,audit_managements,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.446    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,audit_managements#predefined) -> Starting thread
2024-09-06T15:07:25.448    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) - Starting thread
2024-09-06T15:07:25.451 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.494    INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB)
2024-09-06T15:07:25.780    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/chain.crt", "cert_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.crt", "key_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "124794842200288"
2024-09-06T15:07:25.781    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:28.870    INFO InputProcess::CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Setup for module <CortexXdrBasePuller> has been successfully executed

Restart the persistence

Some services in this collector use persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the start_date parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

This Service no longer uses edr.cortex_xdr.alerts_multi_event table to ingest the events Data. Now, All the Data will be ingested in edr.cortex_xdr.alerts_multi .
We are using v2/alerts/get_alerts_multi_events endpoint now but earlier we were using v1/alerts/get_alerts_multi_eventsendpoint to fetch the data.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

note

Note that the PrePull action is executed only one time before the first run of the Pull action.

024-09-06T15:30:53.884    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Pull Started
2024-09-06T15:30:53.886    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> start time_stamp to fetch the data is: 1725616793881 
2024-09-06T15:30:53.887    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 11 messages (60.001491 seconds) => 0 msg/sec
2024-09-06T15:30:59.379    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.380    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Sent 0 events to Devo.
2024-09-06T15:30:59.382    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.383    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.383    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> The data is up to date!

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T15:30:59.382    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in the Data Search area of Devo.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Table edr.cortex_xdr.alerts has been deprecated. Now, the data of incident Alerts (v1/incident/get_incident_alert_data) is ingested in the table edr.cortex_xdr.incident_alert

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

note

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-09-06T17:13:57.587    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Pull Started
2024-09-06T17:13:57.588    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> start time_stamp to fetch the data is: 1724098800000 
2024-09-06T17:13:59.093    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.958.
2024-09-06T17:13:59.094    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Sent 15 events to Devo.
2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.950.
2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.948.
2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> The data is up to date!
2024-09-06T17:13:59.096    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Data collection completed. Elapsed time: 1.518 seconds. Waiting for 58.482 second(s) until the next one

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.948.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Devo categorization and destination

All events of this service are ingested into table edr.cortex_xdr.audit_management

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

note

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-09-06T17:17:05.936    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Pull Started
2024-09-06T17:17:05.939    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> start time_stamp to fetch the data is: 1725623165933 
2024-09-06T17:16:09.137    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.627.
2024-09-06T17:16:09.139    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Sent 2 events to Devo.
2024-09-06T17:16:09.143    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.626.
2024-09-06T17:16:09.145    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.626.
2024-09-06T17:17:09.094    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> The data is up to date!
2024-09-06T17:17:09.094    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Data collection completed. Elapsed time: 3.160 seconds. Waiting for 56.840 second(s) until the next one

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T17:16:09.145    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.626.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Devo categorization and destination

All events of this service are ingested into table edr.cortex_xdr.all_alert

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

note

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-09-06T17:24:44.923    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Pull Started
2024-09-06T17:24:44.925    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> start time_stamp to fetch the data is: 1725308400000 
2024-09-06T17:24:52.766    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.819.
2024-09-06T17:24:52.767    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Sent 77 events to Devo.
2024-09-06T17:24:52.768    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.817.
2024-09-06T17:24:52.768    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.816.
2024-09-06T17:24:52.769    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> The data is up to date!
2024-09-06T17:24:52.770    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Data collection completed. Elapsed time: 7.871 seconds. Waiting for 52.129 second(s) until the next one

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T17:24:52.768    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.816.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module, including all services together:

2024-09-06T15:07:25.139    INFO MainProcess::MainThread -> Started all objects from "MainProcess" process
2024-09-06T15:07:25.144    INFO InputProcess::MainThread -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork")
2024-09-06T15:07:25.220    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gz
2024-09-06T15:07:25.223    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.224    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec3289"
2024-09-06T15:07:25.227    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.232    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.232    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;standard_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/b1ee30ea2c4b3be1eb50c8b2ca80d8d8"
2024-09-06T15:07:25.235    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.236    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.238    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputStandardConsumer;standard_senders;0.json.gz
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/de28663278a264d05d531fbc1db51a93"
2024-09-06T15:07:25.242    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.243    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;lookup_senders;devo_1.json.gz
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/a28c135f3fc69ece87d8fc2fdb029e8c"
2024-09-06T15:07:25.251    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;lookup_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.252    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/df8895fef2a509cbd87fcc9850dc0c81"
2024-09-06T15:07:25.255    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz
2024-09-06T15:07:25.256    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/865a79c1b99ad39b22becc235c9732cb"
2024-09-06T15:07:25.259    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.259    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/e784c94a6fe28de5937e50674407d82e"
2024-09-06T15:07:25.265    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.265    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-09-06T15:07:25.269    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.271    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.272    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/10dd360c86621afd5a28a029a0dddcf6"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.274    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.275    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.275    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.275    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.276    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.277    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.279    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.279    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.280    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.282    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.285    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.286    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.287    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.294    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.298    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.299    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.300    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.301    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables()

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Number of available senders: 1, sender manager internal queue size: 0
2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2024-09-06T15:32:47.811    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Lookup - Total number of messages sent: 0, messages sent since "2024-09-06 09:57:47.810025+00:00": 0 (elapsed 0.000 seconds)

note

By default, these information traces will be displayed every 10 minutes.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

note

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 0, messages sent since "2024-09-06 09:57:47.810025+00:00": 0 (elapsed 0.000 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

0 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2024-09-06 09:57:47.810025+00:00.
0 events where sent to Devo between the last UTC checkpoint and now.
Those 0 events required 0.000 seconds to be delivered.

note

By default these traces will be shown every 10 minutes.

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

Some collectors can be tuned to reduce the memory impact when pulling data in large infrastructures.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
Al metrics (Global, RSS, VMS) include the value before freeing and after: previous -> after freeing memory

2024-09-06T15:07:25.494    INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB)
2024-09-06T15:09:25.375    INFO OutputProcess::MainThread -> [GC] global: 42.4% -> 42.4%, process: RSS(44.46MiB -> 44.46MiB), VMS(929.14MiB -> 929.14MiB)

note

Differences between RSS and VMS memory usage:

RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using

Differences between RSS and VMS memory usage:

RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.

Error type	Error ID	Error message	Cause	Solution
`SetupError`	100	`HTTP Error occurred while checking the server health for cortex from remote source: {e}`	You will get the status code with error msg.	Ensure that the collector has the necessary permissions and proper credentials to access the Cortex xdr API and contact the developer with exact error message
`PullError`	300	`Error while fetching the event data for service {service_name} and error: {e}`	You will get the status code with error msg.	This is an internal issue. Contact with Devo Support team.
`ApiError`	400	`HTTP ERROR 400: Bad request: The server could not understand the request`	Bad Request	Kindly reach out to the developer
	401	`HTTP ERROR 401: Unauthorized: Authentication is required and has failed or has not been provided`	Credentials provided is not correct	Please check and provide the correct credentials
	402	`Unauthorized access. User does not have the required license type to run this API`	User doesn't have the required permission	Kindly get required permission and reach the developer with the exact msg.
	403	`HTTP ERROR 500: Server Error: An error occurred on the server`	Internal server error. An unverified status of API communication type error	Kindly reach the developer with the exact msg.
	404	`Request Error: Received status code {status_code}`	You will get the status code with error msg.	Kindly reach the developer with the exact msg.
	405	`HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.`	Number of request has exceeded for the cortex api	Kindly reach the developer with the exact msg.
	406	`HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.`	Number of request has exceeded for the cortex api	Kindly reach the developer with the exact msg.
	407	`HTTP ERROR 429: Too Many Requests: The user has sent too many requests in a given amount of time.`	Number of request has exceeded for the cortex api	Kindly reach the developer with the exact msg.
	408	`HTTP Error occurred while retrieving events from cortex server, summary: {str(e)}, details: {details_str}`	You will get the status code with error msg	Kindly reach the developer with the exact msg.
	409	`Some error occurred while retrieving events from cortex server. Error details {str(e)}`	You will get the status code with error msg	Kindly reach the developer with the exact msg.

Change log

Release

Released on

Release type

Recommendations

v2.0.3

04 Dec 2024

Recommended version

Improvements

Added 2 new puller and a base puller
Improved unit tests to added some extra tests
Added checks for validating the start_time and if the time is not in future

v2.0.2

07 Nov 2024

Upgrade

Improvements

Updated DCSDK from 1.12.4 to 1.13.1
- Fixed bug related to module_global_status value in message_metrics
- PEP8 Cleanup
- New metric endpoint (http://0.0.0.0:3000/metrics)
- Changed some metric structure that were already sent to Devo before (devo.collector.metric.*)
- Improved MacOS compatibility (for the development phase)
- Updated DevoSDK to version 6.0.0
- Puller and PullerSetup now have the same id structure
- Changed some console log traces to DEBUG
- Fixed bug related to rate_limiter object (the object was not properly internally released)
- Improved Filesystem persistence behavior
- New libraries installed:
  - uvicorn==0.31.1
  - fastapi==0.115.0
  - Updated libraries:
  - jsonschema==4.19.1 -> jsonschema==4.23.0
  - psutil==5.9.6 -> psutil==5.9.8
  - python-dateutil==2.8.2 -> python-dateutil==2.9.0.post0

Bug

Modified paging logic so that the collector responds correctly to controlled stops.

v2.0.1

30 Sep 2024

Upgrade

Improvements

upgraded the docker base image to 1.3.1

Bug fixing

Added condition to resolve the infinite issue

v2.0.0

06 Sep 2024

Upgrade

Improvements

Added new services: audit_management and all_alert.
Modified the table of incident alert.
Removed the usage of alert multi event table.
Using v2 API for alert service.
Unified all the puller services in a single puller.
Moved api_endpoint in the collector definition.

v1.4.0

23 Aug 2024

Upgrade

Improvements:

Added start_time as an optional parameters for both of the services.
Added deduplication logic for both services
Updated Docker image base to version v1.3.0 in Dockerfile
Updated DCSDK from v1.11.1 to v1.12.4
- Added new sender for relay in house + TLS
- Added persistence functionality for gzip sending buffer
- Added Automatic activation of gzip sending
- Improved behaviour when persistence fails
- Upgraded DevoSDK dependency
- Fixed console log encoding
- Restructured python classes
- Improved behaviour with non-utf8 characters
- Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)
- New persistence format/structure (compression in some cases)
- Removed dmesg execution (It was invalid for docker execution)
- Applied changes to make DCSDK compatible with MacOS
- Improved behavior with non-utf8 characters
- Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)
- New persistence format/structure (compression in some cases)
- Removed dmesg execution (It was invalid for docker execution)
- DevoSDK has been updated to version 5.4.0

v1.3.0

23 May 2024

Upgrade

Improvements:

Upgrade DC SDK to the latest version 1.11.1
Upgrade the Docker base image to 1.2.0

v1.2.0

20 Mar 2024

Initial version

Improvements:

Added 'start_time' in config file for alerts service
Added logs