Document toolboxDocument toolbox

Cortex XDR collector

Overview

Cortex XDR delivers enterprise-wide protection by analyzing data from any source to stop sophisticated attacks.

  • Eliminate blind spots with complete visibility

  • Simplify security operations to cut mean time to respond (MTTR)

  • Harness the scale of the cloud for AI and analytics

  • Lower costs by consolidating tools and improving SOC efficiency.

Cortex XDR collector migration guide (from 1.x.x to 2.x.x)

If you need to migrate an old collector version to a more recent one, please check the migration process in this article.

Devo collector features

Feature

Details

Feature

Details

Allow parallel downloading (multipod)

not allowed

Running environments

  • collector server

  • on-premise

Populated Devo events

table

Flattening preprocessing

no

Requires IP Whitelisting

no

Data sources

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Alerts

Get a list of alerts with multiple events.

https://<api-yourfqdn>/public_api/v2/alerts/get_alerts_multi_events

alerts

  • edr.cortex_xdr.alerts_multi

You can override this in override_devo_tag in the incident module definition.

v2.0.0

Incidents

Get Incidents:
Get a list of incidents filtered by a list of incident IDs, modification time, or creation time.

Get Incident Alerts:
Get extra data fields of a specific incident including alerts and key artifacts.

List Incidents:
https://<api-yourfqdn>/public_api/v1/incidents/get_incidents

List Incident Alerts:

https://<api-yourfqdn>/public_api/v1/incidents/get_incident_extra_data

incidents

List Incidents:

  • edr.cortex_xdr.incidents

You can override this in override_devo_tag in the incident module definition.

List Incident Alerts:

  • edr.cortex_xdr.incident_alert

You can override this in override_devo_tag in the incident module definition.

v2.0.0

All Alerts

Get a list of alerts.

https://<api-yourfqdn>/public_api/v1/alerts/get_alerts

all_alerts

edr.cortex_xdr.all_alert

v2.0.0

Audit Managements

Get audit management logs.

https://<api-yourfqdn>/public_api/v1/audits/management_logs

audit_management

  • edr.cortex_xdr.audit_management

v2.0.0

Vendor configuration

Generate API key

  1. Log in to your Cortex XDR console with an administrative account.

  2. Go to the Settings menu (gear icon).

  3. Go to API Keys under the Integrations section.

  4. Click Add Key to generate a new API key.

  5. Select the roles or permissions you want to assign to this key (these determine the actions the key can perform).

  6. Click Create for generating the API Key along with an API Key ID. Save these securely, as you may not be able to retrieve them later.

API FQDN

  • The API FQDN (Fully Qualified Domain Name) can be found in the documentation or directly in the URL when you are logged into your Cortex XDR console.

  • It will usually be in the format: https://<YOUR_REGION>.xdr.paloaltonetworks.com

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.

Setting

Details

Setting

Details

api_key

The API Key is your unique identifier for Cortex XDR.

api_key_id

The API Key ID is your unique token used to authenticate the API key for Cortex XDR.

api_fqdn

API FQDN is a unique host and domain name associated with each tenant for Cortex XDR.

Accepted authentication methods

Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials configuration block.

Authentication method

API key

API key ID

API FQDN

Authentication method

API key

API key ID

API FQDN

API KEY / API KEY ID / API FQDN

required

required

required

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Verify data collection

Service components

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup is common for every service, so its output is displayed in this section.

On the other hand, Pullers are specific to each Service, so their outputs are shown in their corresponding sections below.

Setup output

A successful run has the following output messages for the setup module:

2024-09-06T15:07:25.139 INFO MainProcess::MainThread -> Started all objects from "MainProcess" process 2024-09-06T15:07:25.144 INFO InputProcess::MainThread -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork") 2024-09-06T15:07:25.220 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gz 2024-09-06T15:07:25.223 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.224 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec3289" 2024-09-06T15:07:25.227 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.json 2024-09-06T15:07:25.231 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.231 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164" 2024-09-06T15:07:25.232 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7" 2024-09-06T15:07:25.232 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;standard_senders;devo_sender_0.json.gz 2024-09-06T15:07:25.233 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed 2024-09-06T15:07:25.233 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164" 2024-09-06T15:07:25.233 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7" 2024-09-06T15:07:25.235 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.235 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/b1ee30ea2c4b3be1eb50c8b2ca80d8d8" 2024-09-06T15:07:25.235 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Starting the execution of init_variables() 2024-09-06T15:07:25.236 INFO InputProcess::MainThread -> Validating service metadata 2024-09-06T15:07:25.238 INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputStandardConsumer;standard_senders;0.json.gz 2024-09-06T15:07:25.241 INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.241 INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/de28663278a264d05d531fbc1db51a93" 2024-09-06T15:07:25.242 INFO InputProcess::MainThread -> Validating defined module definition 2024-09-06T15:07:25.243 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;lookup_senders;devo_1.json.gz 2024-09-06T15:07:25.246 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.246 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/a28c135f3fc69ece87d8fc2fdb029e8c" 2024-09-06T15:07:25.251 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;lookup_senders;devo_sender_0.json.gz 2024-09-06T15:07:25.252 INFO InputProcess::MainThread -> Validating common input config 2024-09-06T15:07:25.253 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.253 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/df8895fef2a509cbd87fcc9850dc0c81" 2024-09-06T15:07:25.255 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz 2024-09-06T15:07:25.256 INFO InputProcess::MainThread -> Validating service input config 2024-09-06T15:07:25.257 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.257 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/865a79c1b99ad39b22becc235c9732cb" 2024-09-06T15:07:25.259 INFO InputProcess::MainThread -> Running overriding rules 2024-09-06T15:07:25.259 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz 2024-09-06T15:07:25.260 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user 2024-09-06T15:07:25.260 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead. 2024-09-06T15:07:25.261 INFO InputProcess::MainThread -> Adding raw config to the collector store 2024-09-06T15:07:25.261 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.261 INFO InputProcess::MainThread -> Running custom validation rules 2024-09-06T15:07:25.261 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/e784c94a6fe28de5937e50674407d82e" 2024-09-06T15:07:25.265 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables() 2024-09-06T15:07:25.265 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz 2024-09-06T15:07:25.268 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.268 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b" 2024-09-06T15:07:25.269 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.json 2024-09-06T15:07:25.271 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz 2024-09-06T15:07:25.272 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.272 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f" 2024-09-06T15:07:25.272 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.272 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174" 2024-09-06T15:07:25.273 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/10dd360c86621afd5a28a029a0dddcf6" 2024-09-06T15:07:25.273 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread 2024-09-06T15:07:25.274 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed 2024-09-06T15:07:25.274 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f" 2024-09-06T15:07:25.274 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174" 2024-09-06T15:07:25.274 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds) 2024-09-06T15:07:25.275 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread 2024-09-06T15:07:25.275 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables() 2024-09-06T15:07:25.275 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-09-06T15:07:25.276 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-09-06T15:07:25.276 INFO InputProcess::MainThread -> Validating service metadata 2024-09-06T15:07:25.276 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread 2024-09-06T15:07:25.277 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds) 2024-09-06T15:07:25.279 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-09-06T15:07:25.279 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-09-06T15:07:25.279 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread 2024-09-06T15:07:25.279 INFO InputProcess::MainThread -> Validating defined module definition 2024-09-06T15:07:25.280 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-09-06T15:07:25.280 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-09-06T15:07:25.280 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread 2024-09-06T15:07:25.282 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds) 2024-09-06T15:07:25.283 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-09-06T15:07:25.283 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 2024-09-06T15:07:25.283 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-09-06T15:07:25.283 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-09-06T15:07:25.285 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread 2024-09-06T15:07:25.285 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-09-06T15:07:25.285 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 2024-09-06T15:07:25.285 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-09-06T15:07:25.286 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-09-06T15:07:25.287 INFO InputProcess::MainThread -> Validating common input config 2024-09-06T15:07:25.289 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-09-06T15:07:25.289 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-09-06T15:07:25.294 INFO InputProcess::MainThread -> Validating service input config 2024-09-06T15:07:25.298 INFO InputProcess::MainThread -> Running overriding rules 2024-09-06T15:07:25.299 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user 2024-09-06T15:07:25.300 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead. 2024-09-06T15:07:25.301 INFO InputProcess::MainThread -> Adding raw config to the collector store 2024-09-06T15:07:25.302 INFO InputProcess::MainThread -> Running custom validation rules 2024-09-06T15:07:25.302 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables() 2024-09-06T15:07:25.307 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;all_alerts;predefined;CortexXdrBasePuller.json 2024-09-06T15:07:25.310 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.311 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf" 2024-09-06T15:07:25.311 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef" 2024-09-06T15:07:25.312 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed 2024-09-06T15:07:25.313 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf" 2024-09-06T15:07:25.313 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef" 2024-09-06T15:07:25.314 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Starting the execution of init_variables() 2024-09-06T15:07:25.315 INFO InputProcess::MainThread -> Validating service metadata 2024-09-06T15:07:25.321 INFO InputProcess::MainThread -> Validating defined module definition 2024-09-06T15:07:25.339 INFO InputProcess::MainThread -> Validating common input config 2024-09-06T15:07:25.339 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-09-06T15:07:25.340 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.05 2024-09-06T15:07:25.341 INFO OutputProcess::MainThread -> [GC] global: 42.0% -> 42.0%, process: RSS(41.21MiB -> 41.21MiB), VMS(929.01MiB -> 929.01MiB) 2024-09-06T15:07:25.349 INFO InputProcess::MainThread -> Validating service input config 2024-09-06T15:07:25.353 INFO InputProcess::MainThread -> Running overriding rules 2024-09-06T15:07:25.354 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user 2024-09-06T15:07:25.356 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead. 2024-09-06T15:07:25.357 INFO InputProcess::MainThread -> Adding raw config to the collector store 2024-09-06T15:07:25.358 INFO InputProcess::MainThread -> Running custom validation rules 2024-09-06T15:07:25.359 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Finalizing the execution of init_variables() 2024-09-06T15:07:25.367 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;audit_managements;predefined;CortexXdrBasePuller.json 2024-09-06T15:07:25.373 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated 2024-09-06T15:07:25.373 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2" 2024-09-06T15:07:25.373 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc" 2024-09-06T15:07:25.375 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed 2024-09-06T15:07:25.376 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2" 2024-09-06T15:07:25.376 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc" 2024-09-06T15:07:25.378 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Starting the execution of init_variables() 2024-09-06T15:07:25.379 INFO InputProcess::MainThread -> Validating service metadata 2024-09-06T15:07:25.386 INFO InputProcess::MainThread -> Validating defined module definition 2024-09-06T15:07:25.399 INFO InputProcess::MainThread -> Validating common input config 2024-09-06T15:07:25.405 INFO InputProcess::MainThread -> Validating service input config 2024-09-06T15:07:25.408 INFO InputProcess::MainThread -> Running overriding rules 2024-09-06T15:07:25.409 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user 2024-09-06T15:07:25.410 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead. 2024-09-06T15:07:25.411 INFO InputProcess::MainThread -> Adding raw config to the collector store 2024-09-06T15:07:25.413 INFO InputProcess::MainThread -> Running custom validation rules 2024-09-06T15:07:25.415 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Finalizing the execution of init_variables() 2024-09-06T15:07:25.419 INFO InputProcess::MainThread -> InputThread(cortex_xdr,1243598) - Starting thread (execution_period=60s) 2024-09-06T15:07:25.420 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,incidents,predefined) - Starting thread (execution_period=60s) 2024-09-06T15:07:25.421 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Starting thread 2024-09-06T15:07:25.422 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) - Starting thread 2024-09-06T15:07:25.430 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Waiting until setup will be executed 2024-09-06T15:07:25.431 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,alerts,predefined) - Starting thread (execution_period=60s) 2024-09-06T15:07:25.432 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,alerts#predefined) -> Starting thread 2024-09-06T15:07:25.433 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) - Starting thread 2024-09-06T15:07:25.435 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Waiting until setup will be executed 2024-09-06T15:07:25.438 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,all_alerts,predefined) - Starting thread (execution_period=60s) 2024-09-06T15:07:25.440 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,all_alerts#predefined) -> Starting thread 2024-09-06T15:07:25.440 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) - Starting thread 2024-09-06T15:07:25.444 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Waiting until setup will be executed 2024-09-06T15:07:25.445 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,audit_managements,predefined) - Starting thread (execution_period=60s) 2024-09-06T15:07:25.446 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,audit_managements#predefined) -> Starting thread 2024-09-06T15:07:25.448 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) - Starting thread 2024-09-06T15:07:25.451 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Waiting until setup will be executed 2024-09-06T15:07:25.494 INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB) 2024-09-06T15:07:25.780 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/chain.crt", "cert_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.crt", "key_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "124794842200288" 2024-09-06T15:07:25.781 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-09-06T15:07:28.870 INFO InputProcess::CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Setup for module <CortexXdrBasePuller> has been successfully executed

Restart the persistence

Some services in this collector use persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

  1. Edit the configuration file.

  2. Change the value of the start_date parameter to a different one.

  3. Save the changes.

  4. Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

 

 

 

 

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

024-09-06T15:30:53.884 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Pull Started 2024-09-06T15:30:53.886 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> start time_stamp to fetch the data is: 1725616793881 2024-09-06T15:30:53.887 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 11 messages (60.001491 seconds) => 0 msg/sec 2024-09-06T15:30:59.379 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. 2024-09-06T15:30:59.380 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Sent 0 events to Devo. 2024-09-06T15:30:59.382 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. 2024-09-06T15:30:59.383 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. 2024-09-06T15:30:59.383 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> The data is up to date!

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T15:30:59.382 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Devo categorization and destination

All events of this service are ingested into table edr.cortex_xdr.audit_management

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Devo categorization and destination

All events of this service are ingested into table edr.cortex_xdr.all_alert

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Change log

Release

Released on

Release type

Recommendations

Release

Released on

Release type

Recommendations

v2.0.3

Dec 4, 2024

IMPROVEMENT

Recommended version

v2.0.2

Nov 7, 2024

IMPROVEMENT
bug fixing

Upgrade

 

v2.0.1

Sep 30, 2024

bug fixing

IMPROVEMENT

Upgrade

v2.0.0

Sep 6, 2024

bug fixing

IMPROVEMENT

Upgrade

v1.4.0

Aug 23, 2024

IMPROVEMENT

Upgrade

v1.3.0

May 23, 2024

IMPROVEMENT

Upgrade

v1.2.0

Mar 20, 2024

IMPROVEMENT

 Initial version