Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

Overview

Cortex XDR delivers enterprise-wide protection by analyzing data from any source to stop sophisticated attacks.

  • Eliminate blind spots with complete visibility

  • Simplify security operations to cut mean time to respond (MTTR)

  • Harness the scale of the cloud for AI and analytics

  • Lower costs by consolidating tools and improving SOC efficiency.

Cortex XDR collector migration guide (from 1.x.x to 2.x.x)

If you need to migrate an old collector version to a more recent one, please check the migration process in this article.

Devo collector features

Feature

Details

Allow parallel downloading (multipod)

not allowed

Running environments

  • collector server

  • on-premise

Populated Devo events

table

Flattening preprocessing

no

Requires IP Whitelisting

no

Data sources

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Alerts

Get a list of alerts with multiple events.

https://<api-yourfqdn>/public_api/v2/alerts/get_alerts_multi_events

alerts

  • edr.cortex_xdr.alerts_multi

You can override this in override_devo_tag in the incident module definition.

v2.0.0

Incidents

Get Incidents:
Get a list of incidents filtered by a list of incident IDs, modification time, or creation time.

Get Incident Alerts:
Get extra data fields of a specific incident including alerts and key artifacts.

List Incidents:
https://<api-yourfqdn>/public_api/v1/incidents/get_incidents

List Incident Alerts:

https://<api-yourfqdn>/public_api/v1/incidents/get_incident_extra_data

incidents

List Incidents:

  • edr.cortex_xdr.incidents

You can override this in override_devo_tag in the incident module definition.

List Incident Alerts:

  • edr.cortex_xdr.incident_alert

You can override this in override_devo_tag in the incident module definition.

v2.0.0

All Alerts

Get a list of alerts.

https://<api-yourfqdn>/public_api/v1/alerts/get_alerts

all_alerts

edr.cortex_xdr.all_alert

You can override this in override_devo_tag in the incident module definition.

v2.0.0

Audit Managements

Get audit management logs.

https://<api-yourfqdn>/public_api/v1/audits/management_logs

audit_management

  • edr.cortex_xdr.audit_management

You can override this in override_devo_tag in the incident module definition.

v2.0.0

Vendor configuration

Generate API key

  1. Log in to your Cortex XDR console with an administrative account.

  2. Go to the Settings menu (gear icon).

  3. Go to API Keys under the Integrations section.

  4. Click Add Key to generate a new API key.

  5. Select the roles or permissions you want to assign to this key (these determine the actions the key can perform).

  6. Click Create for generating the API Key along with an API Key ID. Save these securely, as you may not be able to retrieve them later.

API FQDN

  • The API FQDN (Fully Qualified Domain Name) can be found in the documentation or directly in the URL when you are logged into your Cortex XDR console.

  • It will usually be in the format: https://<YOUR_REGION>.xdr.paloaltonetworks.com

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

api_key

The API Key is your unique identifier for Cortex XDR.

api_key_id

The API Key ID is your unique token used to authenticate the API key for Cortex XDR.

api_fqdn

API FQDN is a unique host and domain name associated with each tenant for Cortex XDR.

See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Accepted authentication methods

Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials configuration block.

Authentication method

API key

API key ID

API FQDN

API KEY / API KEY ID / API FQDN

REQUIRED

REQUIRED

REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

 Common sections for Services

Verify data collection

Service components

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup is common for every service, so its output is displayed in this section.

On the other hand, Pullers are specific to each Service, so their outputs are shown in their corresponding sections below.

Setup output

A successful run has the following output messages for the setup module:

2024-09-06T15:07:25.139    INFO MainProcess::MainThread -> Started all objects from "MainProcess" process
2024-09-06T15:07:25.144    INFO InputProcess::MainThread -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork")
2024-09-06T15:07:25.220    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gz
2024-09-06T15:07:25.223    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.224    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec3289"
2024-09-06T15:07:25.227    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.232    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.232    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;standard_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/b1ee30ea2c4b3be1eb50c8b2ca80d8d8"
2024-09-06T15:07:25.235    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.236    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.238    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputStandardConsumer;standard_senders;0.json.gz
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/de28663278a264d05d531fbc1db51a93"
2024-09-06T15:07:25.242    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.243    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;lookup_senders;devo_1.json.gz
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/a28c135f3fc69ece87d8fc2fdb029e8c"
2024-09-06T15:07:25.251    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;lookup_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.252    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/df8895fef2a509cbd87fcc9850dc0c81"
2024-09-06T15:07:25.255    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz
2024-09-06T15:07:25.256    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/865a79c1b99ad39b22becc235c9732cb"
2024-09-06T15:07:25.259    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.259    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/e784c94a6fe28de5937e50674407d82e"
2024-09-06T15:07:25.265    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.265    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-09-06T15:07:25.269    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.271    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.272    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/10dd360c86621afd5a28a029a0dddcf6"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.274    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.275    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.275    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.275    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.276    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.277    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.279    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.279    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.280    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.282    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.285    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.286    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.287    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.294    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.298    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.299    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.300    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.301    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.307    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;all_alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.310    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.311    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf"
2024-09-06T15:07:25.311    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef"
2024-09-06T15:07:25.312    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.313    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf"
2024-09-06T15:07:25.313    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef"
2024-09-06T15:07:25.314    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.315    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.321    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.339    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.339    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.340    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.05
2024-09-06T15:07:25.341    INFO OutputProcess::MainThread -> [GC] global: 42.0% -> 42.0%, process: RSS(41.21MiB -> 41.21MiB), VMS(929.01MiB -> 929.01MiB)
2024-09-06T15:07:25.349    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.353    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.354    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.356    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.357    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.358    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.359    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.367    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;audit_managements;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.373    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.373    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2"
2024-09-06T15:07:25.373    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc"
2024-09-06T15:07:25.375    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.376    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2"
2024-09-06T15:07:25.376    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc"
2024-09-06T15:07:25.378    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.379    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.386    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.399    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.405    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.408    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.409    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.410    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.411    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.413    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.415    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.419    INFO InputProcess::MainThread -> InputThread(cortex_xdr,1243598) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.420    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,incidents,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.421    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Starting thread
2024-09-06T15:07:25.422    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) - Starting thread
2024-09-06T15:07:25.430 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.431    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,alerts,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.432    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,alerts#predefined) -> Starting thread
2024-09-06T15:07:25.433    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) - Starting thread
2024-09-06T15:07:25.435 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.438    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,all_alerts,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.440    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,all_alerts#predefined) -> Starting thread
2024-09-06T15:07:25.440    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) - Starting thread
2024-09-06T15:07:25.444 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.445    INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,audit_managements,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.446    INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,audit_managements#predefined) -> Starting thread
2024-09-06T15:07:25.448    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) - Starting thread
2024-09-06T15:07:25.451 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.494    INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB)
2024-09-06T15:07:25.780    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/chain.crt", "cert_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.crt", "key_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "124794842200288"
2024-09-06T15:07:25.781    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:28.870    INFO InputProcess::CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Setup for module <CortexXdrBasePuller> has been successfully executed

Restart the persistence

Some services in this collector use persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

  1. Edit the configuration file.

  2. Change the value of the start_date parameter to a different one.

  3. Save the changes.

  4. Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

 Alerts Service

  • This Service no longer uses edr.cortex_xdr.alerts_multi_event table to ingest the events Data. Now, All the Data will be ingested in edr.cortex_xdr.alerts_multi .

  • We are using v2/alerts/get_alerts_multi_events endpoint now but earlier we were using v1/alerts/get_alerts_multi_eventsendpoint to fetch the data.

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

024-09-06T15:30:53.884    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Pull Started
2024-09-06T15:30:53.886    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> start time_stamp to fetch the data is: 1725616793881 
2024-09-06T15:30:53.887    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 11 messages (60.001491 seconds) => 0 msg/sec
2024-09-06T15:30:59.379    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.380    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Sent 0 events to Devo.
2024-09-06T15:30:59.382    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.383    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.383    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> The data is up to date!

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T15:30:59.382    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in the Data Search area of Devo.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

 Incidents Service

Table edr.cortex_xdr.alerts has been deprecated. Now, the data of incident Alerts (v1/incident/get_incident_alert_data) is ingested in the table edr.cortex_xdr.incident_alert

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-09-06T17:13:57.587    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Pull Started
2024-09-06T17:13:57.588    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> start time_stamp to fetch the data is: 1724098800000 
2024-09-06T17:13:59.093    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.958.
2024-09-06T17:13:59.094    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Sent 15 events to Devo.
2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.950.
2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.948.
2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> The data is up to date!
2024-09-06T17:13:59.096    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Data collection completed. Elapsed time: 1.518 seconds. Waiting for 58.482 second(s) until the next one

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T17:13:59.095    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623037578):Number of requests made: 1; Number of events received: 15; Number of duplicated events filtered out: 0; Number of events generated and sent: 15; Average of events per second: 9.948.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

 Audit manegements Service (New Service)

Devo categorization and destination

All events of this service are ingested into table edr.cortex_xdr.audit_management

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-09-06T17:17:05.936    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Pull Started
2024-09-06T17:17:05.939    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> start time_stamp to fetch the data is: 1725623165933 
2024-09-06T17:16:09.137    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.627.
2024-09-06T17:16:09.139    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Sent 2 events to Devo.
2024-09-06T17:16:09.143    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.626.
2024-09-06T17:16:09.145    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.626.
2024-09-06T17:17:09.094    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> The data is up to date!
2024-09-06T17:17:09.094    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Data collection completed. Elapsed time: 3.160 seconds. Waiting for 56.840 second(s) until the next one

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T17:16:09.145    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623165933):Number of requests made: 1; Number of events received: 2; Number of duplicated events filtered out: 0; Number of events generated and sent: 2; Average of events per second: 0.626.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

 All Alerts Service (New Service)

Devo categorization and destination

All events of this service are ingested into table edr.cortex_xdr.all_alert

Verify data collection

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-09-06T17:24:44.923    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Pull Started
2024-09-06T17:24:44.925    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> start time_stamp to fetch the data is: 1725308400000 
2024-09-06T17:24:52.766    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.819.
2024-09-06T17:24:52.767    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Sent 77 events to Devo.
2024-09-06T17:24:52.768    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.817.
2024-09-06T17:24:52.768    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.816.
2024-09-06T17:24:52.769    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> The data is up to date!
2024-09-06T17:24:52.770    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Data collection completed. Elapsed time: 7.871 seconds. Waiting for 52.129 second(s) until the next one

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

2024-09-06T17:24:52.768    INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725623684899):Number of requests made: 1; Number of events received: 77; Number of duplicated events filtered out: 0; Number of events generated and sent: 77; Average of events per second: 9.816.

The @devo_pulling_id value is injected into each event to allow grouping all events ingested by the same pull action. You can use it to get the exact events downloaded on that Pull action in Loxcope.

Restart the persistence

This service makes use of persistence. You can check how to restart it above.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

 Verify collector operations

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

 Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module, including all services together:

2024-09-06T15:07:25.139    INFO MainProcess::MainThread -> Started all objects from "MainProcess" process
2024-09-06T15:07:25.144    INFO InputProcess::MainThread -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork")
2024-09-06T15:07:25.220    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gz
2024-09-06T15:07:25.223    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.224    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec3289"
2024-09-06T15:07:25.227    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.231    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.232    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.232    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;standard_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.233    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.235    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/b1ee30ea2c4b3be1eb50c8b2ca80d8d8"
2024-09-06T15:07:25.235    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.236    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.238    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputStandardConsumer;standard_senders;0.json.gz
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.241    INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/de28663278a264d05d531fbc1db51a93"
2024-09-06T15:07:25.242    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.243    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;lookup_senders;devo_1.json.gz
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.246    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/a28c135f3fc69ece87d8fc2fdb029e8c"
2024-09-06T15:07:25.251    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;lookup_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.252    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.253    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/df8895fef2a509cbd87fcc9850dc0c81"
2024-09-06T15:07:25.255    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz
2024-09-06T15:07:25.256    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.257    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/865a79c1b99ad39b22becc235c9732cb"
2024-09-06T15:07:25.259    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.259    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.260    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.261    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.261    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/e784c94a6fe28de5937e50674407d82e"
2024-09-06T15:07:25.265    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.265    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.268    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-09-06T15:07:25.269    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.271    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.272    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/10dd360c86621afd5a28a029a0dddcf6"
2024-09-06T15:07:25.273    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.274    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.274    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.275    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.275    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.275    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276    INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.276    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.277    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.279    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.279    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.279    INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.280    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.282    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.285    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.285    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.286    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.287    INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.289    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.294    INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.298    INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.299    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.300    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.301    INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.302    INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables()
 Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Number of available senders: 1, sender manager internal queue size: 0
2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2024-09-06T15:32:47.810    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2024-09-06T15:32:47.811    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_1) -> Lookup - Total number of messages sent: 0, messages sent since "2024-09-06 09:57:47.810025+00:00": 0 (elapsed 0.000 seconds)

By default, these information traces will be displayed every 10 minutes.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 0, messages sent since "2024-09-06 09:57:47.810025+00:00": 0 (elapsed 0.000 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

  • 0 events were sent to Devo since the collector started.

  • The last checkpoint timestamp was 2024-09-06 09:57:47.810025+00:00.

  • 0 events where sent to Devo between the last UTC checkpoint and now.

  • Those 0 events required 0.000 seconds to be delivered.

By default these traces will be shown every 10 minutes.

 Check memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

Some collectors can be tuned to reduce the memory impact when pulling data in large infrastructures.

  • The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

  • The global pressure of the available memory is displayed in the global value.

  • Al metrics (Global, RSS, VMS) include the value before freeing and after: previous -> after freeing memory

2024-09-06T15:07:25.494    INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB)
2024-09-06T15:09:25.375    INFO OutputProcess::MainThread -> [GC] global: 42.4% -> 42.4%, process: RSS(44.46MiB -> 44.46MiB), VMS(929.14MiB -> 929.14MiB)

Differences between RSS and VMS memory usage:

  • RSS is the Resident Set Size, which is the actual physical memory the process is using

  • VMS is the Virtual Memory Size which is the virtual memory that process is using

 Enable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

  • To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.

  • To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

 Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the common errors for the current services.

Error type

Error ID

Error message

Cause

Solution

SetupError

100

HTTP Error occurred while checking the server health for cortex

If 401 client error, credentials used are not valid.
or issue with the API request process.

Ensure correct credentials are used if 401 client error. Else Contact devo support team

101

Some error occurred while checking the server health for cortex. Error details

The authentication was correct, but returned an unexpected status code.

This is an internal issue. Contact with Devo Support team.

PullError

300

Expected 200 status code, received {status code}

There has been an error upon API request process.

This is an internal issue. Contact with Devo Support team.

301

Expected 200 status code, received {status code}

You will get the status code with error msg.

Kindly reach the develop with the exact msg.

Change log

Release

Released on

Release type

Recommendations

v2.0.1

BUG FIXING

IMPROVEMENT

Recommended version

 Details

Improvements

  • upgraded the docker base image to 1.3.1

Bug fixing

  • Added condition to resolve the infinite issue

v2.0.0

BUG FIXING

IMPROVEMENT

Upgrade

 Details

Improvements

  • Added new services: violations, audit_management and all_alert.

  • Modified the table of incident alert.

  • Removed the usage of alert multi event table.

  • Using v2 API for alert service.

  • Unified all the puller services in a single puller.

  • Moved api_endpoint in the collector definition.

v1.4.0

IMPROVEMENT

Upgrade

 Details

Improvements:

  • Added start_time as an optional parameters for both of the services.

  • Added deduplication logic for both services

  • Updated Docker image base to version v1.3.0 in Dockerfile

  • Updated DCSDK from v1.11.1 to v1.12.4

    • Added new sender for relay in house + TLS

    • Added persistence functionality for gzip sending buffer

    • Added Automatic activation of gzip sending

    • Improved behaviour when persistence fails

    • Upgraded DevoSDK dependency

    • Fixed console log encoding

    • Restructured python classes

    • Improved behaviour with non-utf8 characters

    • Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • Applied changes to make DCSDK compatible with MacOS

    • Improved behavior with non-utf8 characters

    • Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • DevoSDK has been updated to version 5.4.0

v1.3.0

IMPROVEMENT

Upgrade

 Details

Improvements:

  • Upgrade DC SDK to the latest version 1.11.1

  • Upgrade the Docker base image to 1.2.0

v1.2.0

IMPROVEMENT

 Initial version

 Details

Improvements:

  • Added 'start_time' in config file for alerts service

  • Added logs

  • No labels