Cortex XDR collector
- Juan Tomás Alonso Nieto (Deactivated)
- MD Tausif
- Borja Moro Moreno
Overview
Cortex XDR delivers enterprise-wide protection by analyzing data from any source to stop sophisticated attacks.
Eliminate blind spots with complete visibility
Simplify security operations to cut mean time to respond (MTTR)
Harness the scale of the cloud for AI and analytics
Lower costs by consolidating tools and improving SOC efficiency.
Cortex XDR collector migration guide (from 1.x.x to 2.x.x)
If you need to migrate an old collector version to a more recent one, please check the migration process in this article.
Devo collector features
Feature | Details |
---|
Feature | Details |
---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Requires IP Whitelisting |
|
Data sources
Data source | Description | API endpoint | Collector service name | Devo table | Available from release |
---|
Data source | Description | API endpoint | Collector service name | Devo table | Available from release |
---|---|---|---|---|---|
Alerts | Get a list of alerts with multiple events. |
|
|
You can override this in |
|
Incidents | Get Incidents: Get Incident Alerts: | List Incidents: List Incident Alerts:
|
| List Incidents:
You can override this in List Incident Alerts:
You can override this in |
|
All Alerts | Get a list of alerts. |
|
|
|
|
Audit Managements | Get audit management logs. |
|
|
|
|
Vendor configuration
Generate API key
Log in to your Cortex XDR console with an administrative account.
Go to the Settings menu (gear icon).
Go to API Keys under the Integrations section.
Click Add Key to generate a new API key.
Select the roles or permissions you want to assign to this key (these determine the actions the key can perform).
Click Create for generating the API Key along with an API Key ID. Save these securely, as you may not be able to retrieve them later.
API FQDN
The API FQDN (Fully Qualified Domain Name) can be found in the documentation or directly in the URL when you are logged into your Cortex XDR console.
It will usually be in the format:
https://<YOUR_REGION>.xdr.paloaltonetworks.com
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.
Setting | Details |
---|
Setting | Details |
---|---|
| The API Key is your unique identifier for Cortex XDR. |
| The API Key ID is your unique token used to authenticate the API key for Cortex XDR. |
| API FQDN is a unique host and domain name associated with each tenant for Cortex XDR. |
Accepted authentication methods
Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials
configuration block.
Authentication method | API key | API key ID | API FQDN |
---|
Authentication method | API key | API key ID | API FQDN |
---|---|---|---|
API KEY / API KEY ID / API FQDN | required | required | required |
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Verify data collection
Service components
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup is common for every service, so its output is displayed in this section.
On the other hand, Pullers are specific to each Service, so their outputs are shown in their corresponding sections below.
Setup output
A successful run has the following output messages for the setup module:
2024-09-06T15:07:25.139 INFO MainProcess::MainThread -> Started all objects from "MainProcess" process
2024-09-06T15:07:25.144 INFO InputProcess::MainThread -> Process started (pid=19803, ppid=19784, multiprocessing.start_method="fork")
2024-09-06T15:07:25.220 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;standard_senders;devo_1.json.gz
2024-09-06T15:07:25.223 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.224 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5bdbabbd30e1988d6e2b0b74b6ec3289"
2024-09-06T15:07:25.227 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;incidents;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.231 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.231 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.232 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.232 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;standard_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.233 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.233 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/6a3859facec8d617720d39590e169164"
2024-09-06T15:07:25.233 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/c620dfb5bdc3fc1a386e4924e99aa3e7"
2024-09-06T15:07:25.235 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.235 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/b1ee30ea2c4b3be1eb50c8b2ca80d8d8"
2024-09-06T15:07:25.235 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.236 INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.238 INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputStandardConsumer;standard_senders;0.json.gz
2024-09-06T15:07:25.241 INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.241 INFO OutputProcess::MainThread -> OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/de28663278a264d05d531fbc1db51a93"
2024-09-06T15:07:25.242 INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.243 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;lookup_senders;devo_1.json.gz
2024-09-06T15:07:25.246 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.246 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/a28c135f3fc69ece87d8fc2fdb029e8c"
2024-09-06T15:07:25.251 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;lookup_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.252 INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.253 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.253 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/df8895fef2a509cbd87fcc9850dc0c81"
2024-09-06T15:07:25.255 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz
2024-09-06T15:07:25.256 INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.257 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.257 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/865a79c1b99ad39b22becc235c9732cb"
2024-09-06T15:07:25.259 INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.259 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz
2024-09-06T15:07:25.260 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.260 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.261 INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.261 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.261 INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.261 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/e784c94a6fe28de5937e50674407d82e"
2024-09-06T15:07:25.265 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.265 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz
2024-09-06T15:07:25.268 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.268 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-09-06T15:07:25.269 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.271 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-09-06T15:07:25.272 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.272 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.272 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.273 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/10dd360c86621afd5a28a029a0dddcf6"
2024-09-06T15:07:25.273 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.274 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.274 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/50e7b9f5b3320841b223e83e5b156b5f"
2024-09-06T15:07:25.274 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/5a07c27e666ce7c4400ef58d5b89f174"
2024-09-06T15:07:25.274 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.275 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.275 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.275 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.276 INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.276 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.277 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.279 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.279 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.279 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.279 INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.280 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.280 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-09-06T15:07:25.282 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
2024-09-06T15:07:25.283 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.283 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.283 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.285 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
2024-09-06T15:07:25.285 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.285 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-06T15:07:25.285 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.286 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-06T15:07:25.287 INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.289 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.289 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-06T15:07:25.294 INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.298 INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.299 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.300 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.301 INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.302 INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.302 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.307 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;all_alerts;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.310 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.311 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf"
2024-09-06T15:07:25.311 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef"
2024-09-06T15:07:25.312 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.313 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/30e04fb750908b6e56aa65c0390708bf"
2024-09-06T15:07:25.313 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/9e181361d214f7eaac0dcc0853396aef"
2024-09-06T15:07:25.314 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.315 INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.321 INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.339 INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.339 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:25.340 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.05
2024-09-06T15:07:25.341 INFO OutputProcess::MainThread -> [GC] global: 42.0% -> 42.0%, process: RSS(41.21MiB -> 41.21MiB), VMS(929.01MiB -> 929.01MiB)
2024-09-06T15:07:25.349 INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.353 INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.354 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.356 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.357 INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.358 INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.359 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.367 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-cortex-xdr/state/not_used/cortex_xdr;1243598;audit_managements;predefined;CortexXdrBasePuller.json
2024-09-06T15:07:25.373 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-06T15:07:25.373 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2"
2024-09-06T15:07:25.373 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc"
2024-09-06T15:07:25.375 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] Checking if old persisted data must be removed
2024-09-06T15:07:25.376 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 2), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/ae374cefc744c8268f07d37056e660e2"
2024-09-06T15:07:25.376 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> [PERSISTENCE_SYSTEM] No previous persistence file exists (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-cortex-xdr/state/93607031cbbdac722344eb2b8746cfdc"
2024-09-06T15:07:25.378 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Starting the execution of init_variables()
2024-09-06T15:07:25.379 INFO InputProcess::MainThread -> Validating service metadata
2024-09-06T15:07:25.386 INFO InputProcess::MainThread -> Validating defined module definition
2024-09-06T15:07:25.399 INFO InputProcess::MainThread -> Validating common input config
2024-09-06T15:07:25.405 INFO InputProcess::MainThread -> Validating service input config
2024-09-06T15:07:25.408 INFO InputProcess::MainThread -> Running overriding rules
2024-09-06T15:07:25.409 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2024-09-06T15:07:25.410 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2024-09-06T15:07:25.411 INFO InputProcess::MainThread -> Adding raw config to the collector store
2024-09-06T15:07:25.413 INFO InputProcess::MainThread -> Running custom validation rules
2024-09-06T15:07:25.415 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) Finalizing the execution of init_variables()
2024-09-06T15:07:25.419 INFO InputProcess::MainThread -> InputThread(cortex_xdr,1243598) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.420 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,incidents,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.421 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Starting thread
2024-09-06T15:07:25.422 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) - Starting thread
2024-09-06T15:07:25.430 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,incidents#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.431 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,alerts,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.432 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,alerts#predefined) -> Starting thread
2024-09-06T15:07:25.433 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) - Starting thread
2024-09-06T15:07:25.435 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.438 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,all_alerts,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.440 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,all_alerts#predefined) -> Starting thread
2024-09-06T15:07:25.440 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) - Starting thread
2024-09-06T15:07:25.444 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,all_alerts#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.445 INFO InputProcess::MainThread -> ServiceThread(cortex_xdr,1243598,audit_managements,predefined) - Starting thread (execution_period=60s)
2024-09-06T15:07:25.446 INFO InputProcess::MainThread -> CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,audit_managements#predefined) -> Starting thread
2024-09-06T15:07:25.448 INFO InputProcess::MainThread -> CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) - Starting thread
2024-09-06T15:07:25.451 WARNING InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,audit_managements#predefined) -> Waiting until setup will be executed
2024-09-06T15:07:25.494 INFO InputProcess::MainThread -> [GC] global: 42.1% -> 42.1%, process: RSS(41.61MiB -> 41.73MiB), VMS(1.05GiB -> 1.05GiB)
2024-09-06T15:07:25.780 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/chain.crt", "cert_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.crt", "key_path": "/home/md_tausif/gitlab/devo-collector-cortex-xdr/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "124794842200288"
2024-09-06T15:07:25.781 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-06T15:07:28.870 INFO InputProcess::CortexXdrPullerSetup(cortex_xdr,cortex_xdr#1243598,incidents#predefined) -> Setup for module <CortexXdrBasePuller> has been successfully executed
Restart the persistence
Some services in this collector use persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_date
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Verify data collection
Puller output
A successful initial run has the following output messages for the puller module:
024-09-06T15:30:53.884 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Pull Started
2024-09-06T15:30:53.886 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> start time_stamp to fetch the data is: 1725616793881
2024-09-06T15:30:53.887 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 11 messages (60.001491 seconds) => 0 msg/sec
2024-09-06T15:30:59.379 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.380 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Sent 0 events to Devo.
2024-09-06T15:30:59.382 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.383 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-06T15:30:59.383 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> The data is up to date!
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
2024-09-06T15:30:59.382 INFO InputProcess::CortexXdrBasePuller(cortex_xdr#1243598,alerts#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1725616853882):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
Restart the persistence
This service makes use of persistence. You can check how to restart it above.
Verify data collection
Puller output
A successful initial run has the following output messages for the puller module:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistence
This service makes use of persistence. You can check how to restart it above.
Devo categorization and destination
All events of this service are ingested into table edr.cortex_xdr.audit_management
Verify data collection
Puller output
A successful initial run has the following output messages for the puller module:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistence
This service makes use of persistence. You can check how to restart it above.
Devo categorization and destination
All events of this service are ingested into table edr.cortex_xdr.all_alert
Verify data collection
Puller output
A successful initial run has the following output messages for the puller module:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Restart the persistence
This service makes use of persistence. You can check how to restart it above.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Change log
Release | Released on | Release type | Recommendations |
---|
Release | Released on | Release type | Recommendations |
---|---|---|---|
| Dec 4, 2024 | IMPROVEMENT |
|
| Nov 7, 2024 | IMPROVEMENT |
|
| |||
| Sep 30, 2024 | bug fixing IMPROVEMENT |
|
| Sep 6, 2024 | bug fixing IMPROVEMENT |
|
| Aug 23, 2024 | IMPROVEMENT |
|
| May 23, 2024 | IMPROVEMENT |
|
| Mar 20, 2024 | IMPROVEMENT | |