- Created by Former user, last modified by Juan Tomás Alonso Nieto on May 18, 2023
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 32 Next »
Overview
Malwarebytes Nebula is a cloud-hosted security operations platform that allows you to manage control of any malware or ransomware incident
Devo collector features
Feature | Details |
---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
Configuration requirements
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting | Details |
---|---|
| Credential client ID. |
| Credential client secret. |
| Credential account ID. |
| Credential API base url. |
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.
Data sources
Data Source | Description | API Endpoint | Collector service name | Devo Table | Available from release |
Notifications | Malwarebytes Nebula can notify you when certain events occur, such as when real-time protection or scheduled scans detect threats, or if a new endpoint registers to your console. |
| notifications |
| v1.0.0 |
Detection | The Detections section in Malwarebytes Nebula displays information on all threats, and potential threats, with the action taken for each item found on endpoints in your environment |
| detections |
| v1.0.0 |
Events | Event is a general term for a threat that has occurred, remediation or other action taken on a threat, and other endpoint-related activity. |
| events |
| v1.0.0 |
Vulnerability Management | shows vulnerabilities for installed software and operating systems on managed endpoints. |
| vulnerability_management |
| v1.0.0 |
Suspicious activity | Suspicious Activity Monitoring is a feature included in Malwarebytes Endpoint Detection and Response |
| suspicious_activity |
| v1.0.0 |
DNS Logs Data | Logs of Dns data |
| dns_log_data |
| v1.0.0 |
For more information on how the events are parsed, visit our page.
Vendor setup
There are some steps you need to follow to run the collector.
Accepted authentication methods
Authentication Method | Username | Password |
| REQUIRED | REQUIRED |
| REQUIRED | REQUIRED |
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Events service
Data is first pulled for events based on the historic date provided or default historic days, event_id of the last event is stored in a state file, and data is sorted manually in descending order, the last event will be the old one. New data is compared with the previously stored event id to identify the duplicate items and removed them.
All events of Events service are ingested into the table my.app.nebula.events
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
INFO InputProcess::MainThread -> NebulaEventsDataPuller(example_input,12345,events,predefined) - Starting thread 2023-01-23T16:16:31.386 WARNING InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Waiting until setup will be executed 2023-01-23T16:16:31.386 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Token has expired. Generating the new one 2023-01-23T16:16:31.387 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> The token/header/authentication is expired and it needs to be refreshed 2023-01-23T16:16:31.388 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Requesting access token from the Nebula server 2023-01-23T16:16:31.402 INFO OutputProcess::MainThread -> [GC] global: 25.0% -> 25.0%, process: RSS(46.83MiB -> 47.60MiB), VMS(1.19GiB -> 1.19GiB) 2023-01-23T16:16:31.408 INFO InputProcess::MainThread -> [GC] global: 25.0% -> 25.0%, process: RSS(46.96MiB -> 47.29MiB), VMS(791.23MiB -> 791.48MiB) 2023-01-23T16:16:31.720 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140563744962544" 2023-01-23T16:16:31.721 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140563744962400" 2023-01-23T16:16:32.343 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Requesting access token from the Nebula server 2023-01-23T16:16:32.344 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 16:46:31 2023-01-23T16:16:32.344 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 2023-01-23T16:16:32.344 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Setup for module <NebulaEventsDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
023-01-24T08:03:26.575 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Pull Started 2023-01-24T08:03:27.586 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/events?start=2023-01-24T02:32:26Z 2023-01-24T08:03:27.588 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Removing the duplicate events if present... 2023-01-24T08:03:27.589 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Number of events sent to Devo: 0 2023-01-24T08:03:27.589 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Total number of events: 0 2023-01-24T08:03:27.590 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> State last_polled_timestamp is updated with retrieving timestamp 2023-01-24T08:03:27.591 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Saved state: {'last_polled_timestamp': 1674527606.575356, 'historic_date_utc': None, 'ids_with_same_timestamp': ['0fa33de2-963a-4b7f-b709-4111eb82712c'], '@persistence_version': 1} 2023-01-24T08:03:27.591 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527606575):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. 2023-01-24T08:03:27.593 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> The data is up to date! 2023-01-24T08:03:27.595 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Data collection completed. Elapsed time: 1.019 seconds. Waiting for 58.980 second(s) until the next one
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2023-01-24T08:03:27.591 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527606575):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. 2023-01-24T08:03:27.593 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> The data is up to date!
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
historical_date_utc
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Vulnerability management service
Data is first polled for vulnerability_id based on the historic data provided or default historic days,
vulnerability_id
of the last vulnerability is stored in a state file, and data is sorted manually in descending order, the last vulnerability will be the old one. New data is compared with the previously storedvulnerability_id
to identify the duplicate items and removed them.Based on each vulnerability_id a description of the vulnerability is obtained in the next API call, after removing duplicates for ids.
All events of Vulnerability service are ingested into the table
my.app.nebula.vulnerability_management
.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-01-23T17:09:18.002 INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s) 2023-01-23T17:09:18.002 INFO InputProcess::MainThread -> ServiceThread(example_input,12345,vulnerability_management,predefined) - Starting thread (execution_period=60s) 2023-01-23T17:09:18.002 INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Starting thread 2023-01-23T17:09:18.003 INFO InputProcess::MainThread -> NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) - Starting thread 2023-01-23T17:09:18.003 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Token has expired. Generating the new one 2023-01-23T17:09:18.004 WARNING InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Waiting until setup will be executed 2023-01-23T17:09:18.004 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> The token/header/authentication is expired and it needs to be refreshed 2023-01-23T17:09:18.005 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server 2023-01-23T17:09:18.020 INFO OutputProcess::MainThread -> [GC] global: 25.8% -> 25.9%, process: RSS(46.42MiB -> 48.71MiB), VMS(1.19GiB -> 1.19GiB) 2023-01-23T17:09:18.029 INFO InputProcess::MainThread -> [GC] global: 25.9% -> 25.9%, process: RSS(47.31MiB -> 47.38MiB), VMS(791.48MiB -> 791.48MiB) 2023-01-23T17:09:18.341 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332628086400" 2023-01-23T17:09:18.344 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332642608512" 2023-01-23T17:09:19.010 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server 2023-01-23T17:09:19.011 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 17:39:18 2023-01-23T17:09:19.012 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 2023-01-23T17:09:19.012 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Setup for module <NebulaVulnerabilityDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
2023-01-23T17:19:40.513 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Pull Started 2023-01-23T17:19:41.573 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/export 2023-01-23T17:19:41.574 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Received 5 CVE ids from Nebula Server 2023-01-23T17:19:41.575 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Removing the duplicate cve if present... 2023-01-23T17:19:41.575 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2022-34716'} 2023-01-23T17:19:42.498 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2022-34716 2023-01-23T17:19:42.499 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2022-24464'} 2023-01-23T17:19:43.419 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2022-24464 2023-01-23T17:19:43.419 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2020-8927'} 2023-01-23T17:19:44.393 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2020-8927 2023-01-23T17:19:44.395 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2021-34485'} 2023-01-23T17:19:45.339 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2021-34485 2023-01-23T17:19:45.341 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2021-26423'} 2023-01-23T17:19:46.356 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2021-26423 2023-01-23T17:19:46.359 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Number of vulnerabilities sent to Devo: 5 2023-01-23T17:19:46.361 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> State last_polled_timestamp is updated with retrieving timestamp 2023-01-23T17:19:46.361 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Saved state: {'last_polled_timestamp': 1674474580.484891, 'historic_date_utc': 1669991553.0, 'ids_with_same_timestamp': ['CVE-2021-26423'], '@persistence_version': 1} 2023-01-23T17:19:46.361 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674474580484):Number of requests made: 1; Number of events received: 5; Number of duplicated events filtered out: 0; Number of events generated and sent: 5; Average of events per second: 0.855. 2023-01-23T17:19:46.362 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> The data is up to date! 2023-01-23T17:19:46.363 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Data collection completed. Elapsed time: 5.879 seconds. Waiting for 594.121 second(s) until the next one
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2023-01-23T17:19:46.361 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674474580484):Number of requests made: 1; Number of events received: 5; Number of duplicated events filtered out: 0; Number of events generated and sent: 5; Average of events per second: 0.855. 2023-01-23T17:19:46.362 INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> The data is up to date!
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
historical_date_utc
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Notifications service
All of the notifications will be fetched from the API call, then based on the historic date provided or default historic days we will return notifications with a timestamp higher than the most recent timestamp for that Notification, we will update the state with the id’s and its log time which are polled after removing duplicates.
Hence in the next pull the ids present in the state file will not be pulled again and duplicates will be removed.
All notifications of Notifications service are ingested into the table my.app.nebula.notifications
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-01-23T17:09:18.002 INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s) 2023-01-23T17:09:18.002 INFO InputProcess::MainThread -> ServiceThread(example_input,12345,vulnerability_management,predefined) - Starting thread (execution_period=60s) 2023-01-23T17:09:18.002 INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Starting thread 2023-01-23T17:09:18.003 INFO InputProcess::MainThread -> NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) - Starting thread 2023-01-23T17:09:18.003 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Token has expired. Generating the new one 2023-01-23T17:09:18.004 WARNING InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Waiting until setup will be executed 2023-01-23T17:09:18.004 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> The token/header/authentication is expired and it needs to be refreshed 2023-01-23T17:09:18.005 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server 2023-01-23T17:09:18.020 INFO OutputProcess::MainThread -> [GC] global: 25.8% -> 25.9%, process: RSS(46.42MiB -> 48.71MiB), VMS(1.19GiB -> 1.19GiB) 2023-01-23T17:09:18.029 INFO InputProcess::MainThread -> [GC] global: 25.9% -> 25.9%, process: RSS(47.31MiB -> 47.38MiB), VMS(791.48MiB -> 791.48MiB) 2023-01-23T17:09:18.341 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332628086400" 2023-01-23T17:09:18.344 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332642608512" 2023-01-23T17:09:19.010 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server 2023-01-23T17:09:19.011 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 17:39:18 2023-01-23T17:09:19.012 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 2023-01-23T17:09:19.012 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Setup for module <NebulaVulnerabilityDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
2023-01-23T17:45:13.730 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Pull Started 2023-01-23T17:45:15.115 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/notifications/subscriptions 2023-01-23T17:45:15.116 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Removing the duplicate events if present... 2023-01-23T17:45:15.117 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Number of notifications sent to Devo: 12 2023-01-23T17:45:15.117 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Total number of notifications: 12 2023-01-23T17:45:15.118 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> State last_polled_timestamp is updated with retrieving timestamp 2023-01-23T17:45:15.118 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Saved state: {'last_polled_timestamp': 1674476113.707944, 'historic_date_utc': 1666512777.422, 'ids_with_same_timestamp': {'99316cff-92c6-45ee-881b-bb95b18fd1f5': '2022-12-19T12:11:19.183Z', '62ae6d64-cfce-4da2-9617-aea52d64eabb': '2022-12-19T11:55:28.794Z', 'eeec1ead-8ae3-4303-a774-10c5f33292be': '2022-12-16T11:03:09.881Z', '1ce022f6-543f-4f3a-a14e-e91f37360c53': '2022-12-15T07:39:42.715Z', 'b175c54b-5631-4988-b718-48cd5b06d933': '2022-12-15T06:49:24.427Z', '763e40d3-b33b-4872-b8f1-10a08e169989': '2022-12-15T06:34:16.769Z', 'd972ff54-bde5-4512-b354-46f21894bdd6': '2022-12-14T11:13:08.881Z', '8d0db3ba-5466-4c5d-9ad3-3530eb878596': '2022-12-14T08:42:56.454Z', '166106ff-fa40-4607-b12d-44aef22501c2': '2022-12-14T08:41:19.339Z', 'b37b5dbc-b4e9-4392-aa96-de105dd24ad7': '2022-12-06T06:11:36.628Z', '0ad46e4e-df07-438a-8996-a0e08672e926': '2022-12-02T03:48:34.896Z', '7ee4c548-9f31-4439-a65d-91e31b713ad4': '2022-11-23T08:12:57.422Z'}, '@persistence_version': 1} 2023-01-23T17:45:15.118 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674476113707):Number of requests made: 1; Number of events received: 12; Number of duplicated events filtered out: 0; Number of events generated and sent: 12; Average of events per second: 8.646. 2023-01-23T17:45:15.118 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> The data is up to date! 2023-01-23T17:45:15.119 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Data collection completed. Elapsed time: 1.411 seconds. Waiting for 8.589 second(s) until the next one
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Statistics for this pull cycle (@devo_pulling_id=1674476113707):Number of requests made: 1; Number of events received: 12; Number of duplicated events filtered out: 0; Number of events generated and sent: 12; Average of events per second: 8.646. 2023-01-23T17:45:15.118 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> The data is up to date!
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
historical_date_utc
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Suspicious Activity service
All of the notifications will be fetched from the API call, then based on the historic date provided or default historic days we will return notifications with a timestamp higher than the most recent timestamp for that Notification, we will update the state with the id’s and its log time which are polled after removing duplicates.
Hence in the next pull the ids present in the state file will not be pulled again and duplicates will be removed.
All notifications of Notifications service are ingested into the table my.app.nebula.notifications.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_eu_1) -> Starting thread 2023-01-23T17:45:11.730 INFO InputProcess::MainThread -> [GC] global: 28.3% -> 28.4%, process: RSS(46.75MiB -> 46.99MiB), VMS(791.24MiB -> 791.24MiB) 2023-01-23T17:45:11.751 INFO OutputProcess::MainThread -> [GC] global: 28.4% -> 28.4%, process: RSS(49.03MiB -> 49.44MiB), VMS(1.19GiB -> 1.19GiB) 2023-01-23T17:45:12.477 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "139635964562208" 2023-01-23T17:45:12.478 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "139635964560816" 2023-01-23T17:45:13.273 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Requesting access token from the Nebula server 2023-01-23T17:45:13.276 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 18:15:11 2023-01-23T17:45:13.277 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 2023-01-23T17:45:13.278 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Setup for module <NebulaNotificationsDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Starting data collection every 600 seconds 2023-01-23T22:23:03.544 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Pull Started 2023-01-23T22:23:05.259 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/sa 2023-01-23T22:23:05.264 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Removing the duplicate events if present... 2023-01-23T22:23:05.264 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Number of suspicious_activity sent to Devo: 0 2023-01-23T22:23:05.264 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Total number of suspicious_activity: 1 2023-01-23T22:23:05.265 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> State last_polled_timestamp is updated with retrieving timestamp 2023-01-23T22:23:05.265 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Saved state: {'last_polled_timestamp': 1674492783.536124, 'historic_date_utc': None, 'ids_with_same_timestamp': {}, '@persistence_version': 1} 2023-01-23T22:23:05.265 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674492783536):Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000. 2023-01-23T22:23:05.265 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> The data is up to date! 2023-01-23T22:23:05.265 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Data collection completed. Elapsed time: 1.730 seconds. Waiting for 598.270 second(s) until the next one
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Statistics for this pull cycle (@devo_pulling_id=1674492783536):Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000. 2023-01-23T22:23:05.265 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> The data is up to date!
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
historical_date_utc
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Detection service
All of the detections will be fetched from the API call. We will check if the ID is included in the state file after we get the list of detections. Then based on the historic date provided or default historic days we will return detection with a timestamp higher than the most recent timestamp for that detection, we will update the state with the IDs and their log time which are polled after removing duplicates.
Hence in the next PUll the IDs present in the state file will not be pulled again and duplicates will be removed.
All detection of Detection service is ingested into the table my.app.nebula.detections
.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_eu_1) -> Starting thread 2023-01-24T08:08:45.177 INFO InputProcess::MainThread -> <rate_limiter> setting has been accepted with the content {'period_in_seconds': 60, 'requests_limit_in_units': 25} 2023-01-24T08:08:45.177 WARNING InputProcess::MainThread -> The rate_limiter object has been overridden with the following config: {'period_in_seconds': 60, 'requests_limit_in_units': 25} 2023-01-24T08:08:45.178 INFO InputProcess::MainThread -> Running custom validation rules 2023-01-24T08:08:45.178 INFO InputProcess::MainThread -> Running custom init variables validation 2023-01-24T08:08:45.183 INFO InputProcess::MainThread -> Custom init variables validation completed 2023-01-24T08:08:45.183 INFO InputProcess::MainThread -> NebulaDetectionDataPuller(example_input,12345,detections,predefined) Finalizing the execution of init_variables() 2023-01-24T08:08:45.185 INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s) 2023-01-24T08:08:45.186 INFO InputProcess::MainThread -> ServiceThread(example_input,12345,detections,predefined) - Starting thread (execution_period=60s) 2023-01-24T08:08:45.186 INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Starting thread 2023-01-24T08:08:45.187 INFO InputProcess::MainThread -> NebulaDetectionDataPuller(example_input,12345,detections,predefined) - Starting thread 2023-01-24T08:08:45.187 WARNING InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Waiting until setup will be executed 2023-01-24T08:08:45.188 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Token has expired. Generating the new one 2023-01-24T08:08:45.188 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> The token/header/authentication is expired and it needs to be refreshed 2023-01-24T08:08:45.189 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Requesting access token from the Nebula server 2023-01-24T08:08:45.200 INFO OutputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(46.68MiB -> 48.02MiB), VMS(1.19GiB -> 1.19GiB) 2023-01-24T08:08:45.218 INFO InputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(47.50MiB -> 47.52MiB), VMS(791.47MiB -> 791.47MiB) 2023-01-24T08:08:45.524 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140547371149536" 2023-01-24T08:08:45.532 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140547385450448" 2023-01-24T08:08:46.347 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Requesting access token from the Nebula server 2023-01-24T08:08:46.349 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-24 08:38:45 2023-01-24T08:08:46.350 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 2023-01-24T08:08:46.350 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Setup for module <NebulaDetectionDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Pull Started 2023-01-24T08:08:50.880 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/export 2023-01-24T08:08:50.883 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Received 650 detection ids from Nebula , fetching information for each detection id.... 2023-01-24T08:08:50.936 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:08:50.937 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fa2d7c5f-ffd3-4ad6-8ab0-b703d30c0807 wait for a while ... 2023-01-24T08:08:52.605 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fa2d7c5f-ffd3-4ad6-8ab0-b703d30c0807 2023-01-24T08:08:52.606 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:08:52.606 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 62039c92-f50a-48c3-bd96-94364d76ec2b wait for a while ... 2023-01-24T08:08:54.247 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/62039c92-f50a-48c3-bd96-94364d76ec2b 2023-01-24T08:08:54.247 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:08:54.248 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 15828462-55a0-4f4c-80ea-596b9b66a2bb wait for a while ... 2023-01-24T08:08:55.719 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/15828462-55a0-4f4c-80ea-596b9b66a2bb 2023-01-24T08:08:55.720 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:08:55.721 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 0397d58f-5815-494d-9ecf-76cc8fdb259d wait for a while ... 2023-01-24T08:08:57.213 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/0397d58f-5815-494d-9ecf-76cc8fdb259d 2023-01-24T08:08:57.214 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:08:57.214 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 6bd1f1fd-7c40-401c-8fc5-e66957dd506f wait for a while ... 2023-01-24T08:08:58.750 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/6bd1f1fd-7c40-401c-8fc5-e66957dd506f 2023-01-24T08:08:58.751 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:08:58.752 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = c61ee947-e1c0-4fb2-b32a-a48c107b71fe wait for a while ... 2023-01-24T08:09:00.286 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/c61ee947-e1c0-4fb2-b32a-a48c107b71fe 2023-01-24T08:09:00.286 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:00.287 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = ee255a83-1c30-49b5-966e-e8baeb7e7c1d wait for a while ... 2023-01-24T08:09:01.721 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/ee255a83-1c30-49b5-966e-e8baeb7e7c1d 2023-01-24T08:09:01.721 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:01.722 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 3b1ff58d-88b6-47df-a2f0-4d4416b0a476 wait for a while ... 2023-01-24T08:09:03.153 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/3b1ff58d-88b6-47df-a2f0-4d4416b0a476 2023-01-24T08:09:03.153 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:03.154 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 3d9295a2-25a9-4f75-89e1-b146249efe71 wait for a while ... 2023-01-24T08:09:04.996 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/3d9295a2-25a9-4f75-89e1-b146249efe71 2023-01-24T08:09:04.996 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:04.997 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 40c6dca2-094a-43df-b339-36b71bd4d548 wait for a while ... 2023-01-24T08:09:06.533 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/40c6dca2-094a-43df-b339-36b71bd4d548 2023-01-24T08:09:06.533 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:06.534 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 20804a4a-6d3a-4c69-b7b6-dc64638f040f wait for a while ... 2023-01-24T08:09:08.171 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/20804a4a-6d3a-4c69-b7b6-dc64638f040f 2023-01-24T08:09:08.171 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:08.172 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = bb59bdb2-9eb3-4250-9380-b97186491063 wait for a while ... 2023-01-24T08:09:09.399 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/bb59bdb2-9eb3-4250-9380-b97186491063 2023-01-24T08:09:09.399 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:09.400 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = ca3b45af-2632-4f1f-a40c-4fdd02dead60 wait for a while ... 2023-01-24T08:09:10.835 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/ca3b45af-2632-4f1f-a40c-4fdd02dead60 2023-01-24T08:09:10.836 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:10.837 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 33139f41-395a-48d4-9949-dec3813cc62b wait for a while ... 2023-01-24T08:09:12.164 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/33139f41-395a-48d4-9949-dec3813cc62b 2023-01-24T08:09:12.164 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:12.165 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8ab05aa3-e732-4506-bf67-bd67bf1e9c8a wait for a while ... 2023-01-24T08:09:13.594 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8ab05aa3-e732-4506-bf67-bd67bf1e9c8a 2023-01-24T08:09:13.594 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:13.594 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 737d64e4-9f7c-4da0-95cc-db7478802979 wait for a while ... 2023-01-24T08:09:14.970 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/737d64e4-9f7c-4da0-95cc-db7478802979 2023-01-24T08:09:14.970 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:14.971 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 93967a0c-edee-425a-ab83-a5f8be36e97e wait for a while ... 2023-01-24T08:09:16.670 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/93967a0c-edee-425a-ab83-a5f8be36e97e 2023-01-24T08:09:16.671 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:16.672 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 0216543f-2a7e-4898-8ac9-ede98c6e3d01 wait for a while ... 2023-01-24T08:09:18.308 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/0216543f-2a7e-4898-8ac9-ede98c6e3d01 2023-01-24T08:09:18.309 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:18.309 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 022ce235-4eda-43a4-bd92-ba6934376554 wait for a while ... 2023-01-24T08:09:19.846 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/022ce235-4eda-43a4-bd92-ba6934376554 2023-01-24T08:09:19.848 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:19.849 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 91834b2d-1c97-469e-8cd7-81385683a41e wait for a while ... 2023-01-24T08:09:21.377 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/91834b2d-1c97-469e-8cd7-81385683a41e 2023-01-24T08:09:21.377 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:21.378 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 53c10041-07da-401b-bfbb-a39671b88d18 wait for a while ... 2023-01-24T08:09:22.814 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/53c10041-07da-401b-bfbb-a39671b88d18 2023-01-24T08:09:22.814 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:22.815 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fb6c8138-9cda-49a3-89fb-dcf20b1133f2 wait for a while ... 2023-01-24T08:09:24.145 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fb6c8138-9cda-49a3-89fb-dcf20b1133f2 2023-01-24T08:09:24.145 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:24.146 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 236fd0f5-69a9-4663-88dd-a7559604bd79 wait for a while ... 2023-01-24T08:09:25.575 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/236fd0f5-69a9-4663-88dd-a7559604bd79 2023-01-24T08:09:25.576 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:25.576 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 51567b63-de25-4a9e-b62e-c404b4fc3119 wait for a while ... 2023-01-24T08:09:27.012 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/51567b63-de25-4a9e-b62e-c404b4fc3119 2023-01-24T08:09:27.013 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:27.013 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 4d4e9dbc-f6a2-44e7-a246-a5f98f36f706 wait for a while ... 2023-01-24T08:09:28.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/4d4e9dbc-f6a2-44e7-a246-a5f98f36f706 2023-01-24T08:09:28.550 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:28.550 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 11cdbe65-d5b9-4051-b2f1-42c7c7fa671f wait for a while ... 2023-01-24T08:09:29.983 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/11cdbe65-d5b9-4051-b2f1-42c7c7fa671f 2023-01-24T08:09:29.984 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:29.985 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 825e5fca-8b97-4108-a4c7-621bc122101b wait for a while ... 2023-01-24T08:09:31.416 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/825e5fca-8b97-4108-a4c7-621bc122101b 2023-01-24T08:09:31.417 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:31.418 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = df1cde2f-6b01-4912-ab66-d73e44d048a5 wait for a while ... 2023-01-24T08:09:32.849 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/df1cde2f-6b01-4912-ab66-d73e44d048a5 2023-01-24T08:09:32.851 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:32.852 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 97074472-38ba-4a2c-933e-56fa4716d369 wait for a while ... 2023-01-24T08:09:34.215 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/97074472-38ba-4a2c-933e-56fa4716d369 2023-01-24T08:09:34.217 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:34.218 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 7c8a7370-1051-4180-9552-5c986d158ff1 wait for a while ... 2023-01-24T08:09:35.615 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/7c8a7370-1051-4180-9552-5c986d158ff1 2023-01-24T08:09:35.616 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:35.616 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f6b390f7-f4fe-4ab9-946e-b4283b8e508b wait for a while ... 2023-01-24T08:09:37.049 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f6b390f7-f4fe-4ab9-946e-b4283b8e508b 2023-01-24T08:09:37.051 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:37.052 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = b0a299c3-a541-487b-ac21-0344f99eb2de wait for a while ... 2023-01-24T08:09:38.385 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/b0a299c3-a541-487b-ac21-0344f99eb2de 2023-01-24T08:09:38.386 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:38.386 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 22020f80-53ec-4095-9c8b-58cdefe53903 wait for a while ... 2023-01-24T08:09:39.812 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/22020f80-53ec-4095-9c8b-58cdefe53903 2023-01-24T08:09:39.813 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:39.813 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 34c2bb2c-1445-46b1-8e62-890f52723478 wait for a while ... 2023-01-24T08:09:41.140 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/34c2bb2c-1445-46b1-8e62-890f52723478 2023-01-24T08:09:41.140 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:41.140 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f358ea21-fd91-4577-a1a7-32dbdae6c2f9 wait for a while ... 2023-01-24T08:09:42.503 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f358ea21-fd91-4577-a1a7-32dbdae6c2f9 2023-01-24T08:09:42.504 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:42.504 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = ba4d297e-6605-4b4e-93e0-dd01cdf78e3f wait for a while ... 2023-01-24T08:09:43.909 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/ba4d297e-6605-4b4e-93e0-dd01cdf78e3f 2023-01-24T08:09:43.911 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:43.912 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 4cb1e887-6893-4aad-92bc-236ad272b143 wait for a while ... 2023-01-24T08:09:45.218 INFO OutputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(49.73MiB -> 49.73MiB), VMS(1.20GiB -> 1.20GiB) 2023-01-24T08:09:45.233 INFO InputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(51.98MiB -> 51.98MiB), VMS(792.35MiB -> 792.35MiB) 2023-01-24T08:09:45.234 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> Consumed messages: 41, total_bytes: 28235 (60.059315 seconds) 2023-01-24T08:09:45.234 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 41 messages (60.061175 seconds) => 0 msg/sec 2023-01-24T08:09:45.445 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/4cb1e887-6893-4aad-92bc-236ad272b143 2023-01-24T08:09:45.446 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:45.446 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 95e4edc2-49e5-4ed5-b65b-1b0c56d039b0 wait for a while ... 2023-01-24T08:09:47.144 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/95e4edc2-49e5-4ed5-b65b-1b0c56d039b0 2023-01-24T08:09:47.145 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:47.145 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 46bc8fa8-801d-4f1a-b43e-6ab90fec9843 wait for a while ... 2023-01-24T08:09:48.520 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/46bc8fa8-801d-4f1a-b43e-6ab90fec9843 2023-01-24T08:09:48.522 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:48.524 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 3b88d4f9-edf6-4aee-bee1-26ef90fd112b wait for a while ... 2023-01-24T08:09:50.053 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/3b88d4f9-edf6-4aee-bee1-26ef90fd112b 2023-01-24T08:09:50.055 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:50.056 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 84f490e9-de8e-4b02-b243-5fd7b84e2d6f wait for a while ... 2023-01-24T08:09:51.692 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/84f490e9-de8e-4b02-b243-5fd7b84e2d6f 2023-01-24T08:09:51.693 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:51.694 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 7cec1216-13b2-49c7-8871-c2342f9f021a wait for a while ... 2023-01-24T08:09:53.130 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/7cec1216-13b2-49c7-8871-c2342f9f021a 2023-01-24T08:09:53.132 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:53.133 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 14ec5bbe-52ca-43c3-a946-4027890863a7 wait for a while ... 2023-01-24T08:09:54.560 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/14ec5bbe-52ca-43c3-a946-4027890863a7 2023-01-24T08:09:54.560 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:54.561 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = c16c6dd8-3afc-4d57-8867-41000062f673 wait for a while ... 2023-01-24T08:09:55.995 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/c16c6dd8-3afc-4d57-8867-41000062f673 2023-01-24T08:09:55.996 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:55.997 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 6e67790c-79fd-41f2-8b45-17a88995d1a2 wait for a while ... 2023-01-24T08:09:57.630 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/6e67790c-79fd-41f2-8b45-17a88995d1a2 2023-01-24T08:09:57.631 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:57.631 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8858d65a-a593-4496-8fc2-a80f612470ac wait for a while ... 2023-01-24T08:09:59.070 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8858d65a-a593-4496-8fc2-a80f612470ac 2023-01-24T08:09:59.071 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:59.071 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = aed74e91-3c84-4990-bf50-d724da00b134 wait for a while ... 2023-01-24T08:10:00.503 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/aed74e91-3c84-4990-bf50-d724da00b134 2023-01-24T08:10:00.504 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:00.504 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 814f10c8-ccd0-4f69-923f-eb5a7111ea07 wait for a while ... 2023-01-24T08:10:01.936 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/814f10c8-ccd0-4f69-923f-eb5a7111ea07 2023-01-24T08:10:01.936 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:01.936 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = d32f5f0d-0046-4045-9016-04d16313a112 wait for a while ... 2023-01-24T08:10:03.473 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/d32f5f0d-0046-4045-9016-04d16313a112 2023-01-24T08:10:03.473 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:03.473 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fe5479f7-ef04-4b1b-915e-76dc98770ee8 wait for a while ... 2023-01-24T08:10:04.905 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fe5479f7-ef04-4b1b-915e-76dc98770ee8 2023-01-24T08:10:04.907 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:04.909 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 364d231e-35cb-48ff-a3d4-28b3da6d97a5 wait for a while ... 2023-01-24T08:10:06.287 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/364d231e-35cb-48ff-a3d4-28b3da6d97a5 2023-01-24T08:10:06.288 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:06.288 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3 wait for a while ... 2023-01-24T08:10:07.778 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3 2023-01-24T08:10:07.778 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:07.779 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 6e67790c-79fd-41f2-8b45-17a88995d1a2 wait for a while ... 2023-01-24T08:09:57.630 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/6e67790c-79fd-41f2-8b45-17a88995d1a2 2023-01-24T08:09:57.631 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:57.631 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8858d65a-a593-4496-8fc2-a80f612470ac wait for a while ... 2023-01-24T08:09:59.070 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8858d65a-a593-4496-8fc2-a80f612470ac 2023-01-24T08:09:59.071 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:09:59.071 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = aed74e91-3c84-4990-bf50-d724da00b134 wait for a while ... 2023-01-24T08:10:00.503 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/aed74e91-3c84-4990-bf50-d724da00b134 2023-01-24T08:10:00.504 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:00.504 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 814f10c8-ccd0-4f69-923f-eb5a7111ea07 wait for a while ... 2023-01-24T08:10:01.936 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/814f10c8-ccd0-4f69-923f-eb5a7111ea07 2023-01-24T08:10:01.936 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:01.936 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = d32f5f0d-0046-4045-9016-04d16313a112 wait for a while ... 2023-01-24T08:10:03.473 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/d32f5f0d-0046-4045-9016-04d16313a112 2023-01-24T08:10:03.473 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:03.473 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fe5479f7-ef04-4b1b-915e-76dc98770ee8 wait for a while ... 2023-01-24T08:10:04.905 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fe5479f7-ef04-4b1b-915e-76dc98770ee8 2023-01-24T08:10:04.907 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:04.909 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 364d231e-35cb-48ff-a3d4-28b3da6d97a5 wait for a while ... 2023-01-24T08:10:06.287 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/364d231e-35cb-48ff-a3d4-28b3da6d97a5 2023-01-24T08:10:06.288 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:06.288 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3 wait for a while ... 2023-01-24T08:10:07.778 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3 2023-01-24T08:10:07.778 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:07.779 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 99f714b6-85f3-41ce-96fa-f1f8aa675a88 wait for a while ... 2023-01-24T08:10:09.308 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/99f714b6-85f3-41ce-96fa-f1f8aa675a88 2023-01-24T08:10:09.309 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:09.309 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = aa7c17cd-940f-4b0f-a494-9e7f2a738a8c wait for a while ... 2023-01-24T08:10:10.744 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/aa7c17cd-940f-4b0f-a494-9e7f2a738a8c 2023-01-24T08:10:10.745 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:10.745 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = d3084825-954b-4b1d-aec9-c02b95a412e8 wait for a while ... 2023-01-24T08:10:12.176 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/d3084825-954b-4b1d-aec9-c02b95a412e8 2023-01-24T08:10:12.176 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:12.176 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8365f56e-eb28-40a1-a14c-4380587f87ef wait for a while ... 2023-01-24T08:10:13.709 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8365f56e-eb28-40a1-a14c-4380587f87ef 2023-01-24T08:10:13.710 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present... 2023-01-24T08:10:13.719 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Number of detections sent to Devo: 56 2023-01-24T08:10:13.721 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> State last_polled_timestamp is updated with retrieving timestamp 2023-01-24T08:10:13.722 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Saved state: {'last_polled_timestamp': 1674527927.190399, 'historic_date_utc': 1674461577.422, 'detection_ids': {'fa2d7c5f-ffd3-4ad6-8ab0-b703d30c0807': '2023-01-24T01:55:12.35726Z', '62039c92-f50a-48c3-bd96-94364d76ec2b': '2023-01-24T01:54:00.435667Z', '15828462-55a0-4f4c-80ea-596b9b66a2bb': '2023-01-24T01:52:48.431457Z', '0397d58f-5815-494d-9ecf-76cc8fdb259d': '2023-01-24T01:32:24.58344Z', '6bd1f1fd-7c40-401c-8fc5-e66957dd506f': '2023-01-24T01:00:29.734536Z', 'c61ee947-e1c0-4fb2-b32a-a48c107b71fe': '2023-01-24T00:49:23.387871Z', 'ee255a83-1c30-49b5-966e-e8baeb7e7c1d': '2023-01-24T00:31:59.350819Z', '3b1ff58d-88b6-47df-a2f0-4d4416b0a476': '2023-01-24T00:30:47.430603Z', '3d9295a2-25a9-4f75-89e1-b146249efe71': '2023-01-24T00:29:35.465086Z', '40c6dca2-094a-43df-b339-36b71bd4d548': '2023-01-24T00:11:01.123421Z', '20804a4a-6d3a-4c69-b7b6-dc64638f040f': '2023-01-23T23:57:41.332863Z', 'bb59bdb2-9eb3-4250-9380-b97186491063': '2023-01-23T23:37:25.151688Z', 'ca3b45af-2632-4f1f-a40c-4fdd02dead60': '2023-01-23T23:14:49.155606Z', '33139f41-395a-48d4-9949-dec3813cc62b': '2023-01-23T23:14:05.365414Z', '8ab05aa3-e732-4506-bf67-bd67bf1e9c8a': '2023-01-23T23:08:50.457342Z', '737d64e4-9f7c-4da0-95cc-db7478802979': '2023-01-23T23:07:39.026314Z', '93967a0c-edee-425a-ab83-a5f8be36e97e': '2023-01-23T23:06:29.931750Z', '0216543f-2a7e-4898-8ac9-ede98c6e3d01': '2023-01-23T22:15:56.204348Z', '022ce235-4eda-43a4-bd92-ba6934376554': '2023-01-23T22:04:35.363259Z', '91834b2d-1c97-469e-8cd7-81385683a41e': '2023-01-23T22:04:19.1193Z', '53c10041-07da-401b-bfbb-a39671b88d18': '2023-01-23T21:40:59.057789Z', 'fb6c8138-9cda-49a3-89fb-dcf20b1133f2': '2023-01-23T21:13:08.316732Z', '236fd0f5-69a9-4663-88dd-a7559604bd79': '2023-01-23T20:50:48.680785Z', '51567b63-de25-4a9e-b62e-c404b4fc3119': '2023-01-23T20:40:56.062104Z', '4d4e9dbc-f6a2-44e7-a246-a5f98f36f706': '2023-01-23T20:35:00.208627Z', '11cdbe65-d5b9-4051-b2f1-42c7c7fa671f': '2023-01-23T20:07:41.059141Z', '825e5fca-8b97-4108-a4c7-621bc122101b': '2023-01-23T19:55:57.170841Z', 'df1cde2f-6b01-4912-ab66-d73e44d048a5': '2023-01-23T19:39:23.335462Z', '97074472-38ba-4a2c-933e-56fa4716d369': '2023-01-23T19:14:50.820054Z', '7c8a7370-1051-4180-9552-5c986d158ff1': '2023-01-23T19:10:20.571464Z', 'f6b390f7-f4fe-4ab9-946e-b4283b8e508b': '2023-01-23T19:02:22.163271Z', 'b0a299c3-a541-487b-ac21-0344f99eb2de': '2023-01-23T18:53:12.388744Z', '22020f80-53ec-4095-9c8b-58cdefe53903': '2023-01-23T18:48:15.121402Z', '34c2bb2c-1445-46b1-8e62-890f52723478': '2023-01-23T17:56:25.103127Z', 'f358ea21-fd91-4577-a1a7-32dbdae6c2f9': '2023-01-23T17:36:35.874122Z', 'ba4d297e-6605-4b4e-93e0-dd01cdf78e3f': '2023-01-23T16:35:09.764540Z', '4cb1e887-6893-4aad-92bc-236ad272b143': '2023-01-23T16:24:38.448483Z', '95e4edc2-49e5-4ed5-b65b-1b0c56d039b0': '2023-01-23T16:04:07.716107Z', '46bc8fa8-801d-4f1a-b43e-6ab90fec9843': '2023-01-23T15:59:29.944425Z', '3b88d4f9-edf6-4aee-bee1-26ef90fd112b': '2023-01-23T15:38:52.531519Z', '84f490e9-de8e-4b02-b243-5fd7b84e2d6f': '2023-01-23T15:34:38.225593Z', '7cec1216-13b2-49c7-8871-c2342f9f021a': '2023-01-23T15:24:50.043742Z', '14ec5bbe-52ca-43c3-a946-4027890863a7': '2023-01-23T15:01:11.03246Z', 'c16c6dd8-3afc-4d57-8867-41000062f673': '2023-01-23T13:58:11.770177Z', '6e67790c-79fd-41f2-8b45-17a88995d1a2': '2023-01-23T13:36:57.367930Z', '8858d65a-a593-4496-8fc2-a80f612470ac': '2023-01-23T13:12:11.576420Z', 'aed74e91-3c84-4990-bf50-d724da00b134': '2023-01-23T13:00:32.356141Z', '814f10c8-ccd0-4f69-923f-eb5a7111ea07': '2023-01-23T12:50:53.842600Z', 'd32f5f0d-0046-4045-9016-04d16313a112': '2023-01-23T11:48:35.971445Z', 'fe5479f7-ef04-4b1b-915e-76dc98770ee8': '2023-01-23T11:42:34.132893Z', '364d231e-35cb-48ff-a3d4-28b3da6d97a5': '2023-01-23T11:38:02.863276Z', 'f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3': '2023-01-23T11:22:28.967188Z', '99f714b6-85f3-41ce-96fa-f1f8aa675a88': '2023-01-23T11:17:27.70895Z', 'aa7c17cd-940f-4b0f-a494-9e7f2a738a8c': '2023-01-23T09:31:37.432437Z', 'd3084825-954b-4b1d-aec9-c02b95a412e8': '2023-01-23T09:05:18.292574Z', '8365f56e-eb28-40a1-a14c-4380587f87ef': '2023-01-23T08:36:38.65638Z'}, '@persistence_version': 1} 2023-01-24T08:10:13.722 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527927190):Number of requests made: 1; Number of events received: 650; Number of duplicated events filtered out: 594; Number of events generated and sent: 56; Average of events per second: 0.647. 2023-01-24T08:10:13.723 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> The data is up to date! 2023-01-24T08:10:13.724 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Data collection completed. Elapsed time: 86.534 seconds. Waiting for 513.466 second(s) until the next one 2023-01-24T08:10:13.729 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 3, total_bytes: 1896 (88.558356 seconds)
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2023-01-24T08:10:13.722 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527927190):Number of requests made: 1; Number of events received: 650; Number of duplicated events filtered out: 594; Number of events generated and sent: 56; Average of events per second: 0.647. 2023-01-24T08:10:13.723 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> The data is up to date!
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
historical_date_utc
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
DNS logs service
All of the detections will be fetched from the API call. We will check if the ID is included in the state file after we get the list of detections. Then based on the historic date provided or default historic days we will return detection with a timestamp higher than the most recent timestamp for that detection, we will update the state with the IDs and their log time which are polled after removing duplicates.
Hence in the next PUll the IDs present in the state file will not be pulled again and duplicates will be removed.
All detection of Detection service is ingested into the table my.app.nebula.detections
.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
I2023-01-24T08:07:45.735 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread 2023-01-24T08:07:45.736 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Starting thread (every 300 seconds) 2023-01-24T08:07:45.736 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_eu_1) -> Starting thread 2023-01-24T08:07:45.738 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread 2023-01-24T08:07:45.738 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Starting thread (every 300 seconds) 2023-01-24T08:07:45.739 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_eu_1) -> Starting thread 2023-01-24T08:07:45.740 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread 2023-01-24T08:07:45.740 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Starting thread (every 300 seconds) 2023-01-24T08:07:45.741 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_eu_1) -> Starting thread 2023-01-24T08:07:45.741 INFO InputProcess::MainThread -> <rate_limiter> setting has been accepted with the content {'period_in_seconds': 60, 'requests_limit_in_units': 25} 2023-01-24T08:07:45.741 WARNING InputProcess::MainThread -> The rate_limiter object has been overridden with the following config: {'period_in_seconds': 60, 'requests_limit_in_units': 25} 2023-01-24T08:07:45.742 INFO InputProcess::MainThread -> Running custom validation rules 2023-01-24T08:07:45.742 INFO InputProcess::MainThread -> Running custom init variables validation 2023-01-24T08:07:45.747 INFO InputProcess::MainThread -> Custom init variables validation completed 2023-01-24T08:07:45.747 INFO InputProcess::MainThread -> NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) Finalizing the execution of init_variables() 2023-01-24T08:07:45.751 INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s) 2023-01-24T08:07:45.751 INFO InputProcess::MainThread -> ServiceThread(example_input,12345,dns_log_data,predefined) - Starting thread (execution_period=60s) 2023-01-24T08:07:45.752 INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Starting thread 2023-01-24T08:07:45.753 INFO InputProcess::MainThread -> NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) - Starting thread 2023-01-24T08:07:45.753 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Token has expired. Generating the new one 2023-01-24T08:07:45.754 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> The token/header/authentication is expired and it needs to be refreshed 2023-01-24T08:07:45.754 WARNING InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Waiting until setup will be executed 2023-01-24T08:07:45.755 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Requesting access token from the Nebula server 2023-01-24T08:07:45.772 INFO OutputProcess::MainThread -> [GC] global: 18.3% -> 18.4%, process: RSS(47.02MiB -> 47.86MiB), VMS(1.19GiB -> 1.19GiB) 2023-01-24T08:07:45.787 INFO InputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(47.08MiB -> 47.17MiB), VMS(791.65MiB -> 791.60MiB) 2023-01-24T08:07:46.087 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140013684891072" 2023-01-24T08:07:46.088 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140013684750560" 2023-01-24T08:07:47.784 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Requesting access token from the Nebula server 2023-01-24T08:07:47.786 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-24 08:37:45 2023-01-24T08:07:47.786 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 2023-01-24T08:07:47.786 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Setup for module <NebulaDnsLogsDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
2023-01-24T08:07:58.464 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/dns 2023-01-24T08:07:58.537 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Total Dns available are = 18527 2023-01-24T08:07:58.537 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Received 2000 dns data from Nebula Server 2023-01-24T08:07:58.537 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Removing the duplicate dns if present... 2023-01-24T08:07:58.545 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> State last_polled_timestamp is updated with retrieving timestamp 2023-01-24T08:07:58.545 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Received 317 dns data from Nebula Server after removing duplicates 2023-01-24T08:07:58.560 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Number of DNS logs sent to Devo: 317 2023-01-24T08:07:58.561 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Saved state: {'last_polled_timestamp': 1674527868.758331, 'historic_date_utc': 1674461577.422, 'next_cursor': 'WzE2NzQxMTUwMDM3NzgsImU2MjI3Y2NkLTE0OTItNDQxOC05MThjLWQ1NTRjNDg1ZWQ5YSJd', 'dns_ids': ['1de0c0c0-e88b-461f-806a-e9514a4a190e', '27ee777d-bf62-4f9b-a437-7359427d1283', 'eb3a915e-e957-4142-9b0a-fc1a6de6cdb3', 'f4e52927-0431-4da6-b05b-4dd08d99912b', 'd194dd03-c501-40c9-87c5-146298491e08', '25fa78a0-c932-4aa6-8f5b-027eca3adc19', '8c85017a-67d9-42ca-8ab0-75b585d349f6', 'eeb1e594-e9dc-42ab-9f4f-9505bd0f1af8', 'b10301d1-baf9-4894-a09d-4073ee16115d', '9873f89d-987f-46d9-980e-338607822957', '7ea98f40-2ec4-4718-a331-983476bcfdc7', '2a479e96-4faa-4ead-a1c9-3ee049c8c4a7', '28cdba37-c277-4203-aaab-4a6b5f975f38', '2b5f36d5-7d09-4c2b-aeb4-c216545f9dc5', '303fa343-dd53-4707-af6d-75054ad8c119', '47a5edae-576c-4d7d-b587-751e719bca60', '566e42c1-297b-470e-9b0b-ade0b49e2d92', 'ba9b9196-1a9d-4b38-9090-3eb8c0c61e02', 'be8d149c-a7b6-4a58-8573-96942f3f6d63', 'd559d57d-8c71-4089-9cca-4deae46f63d3', '5515c005-f916-4dd7-8ca5-42bd80f4cc82', '8d478713-ad1b-4f52-8bbf-1338120ea93d', 'd7f10f1f-7b26-4c11-a118-56b29235e197', '18d509ae-4474-450f-be00-729a11d21f58', '3dcff515-4a03-4b8d-80c0-9c089e1fc071', 'b9d9faed-4e1e-46b5-b8f0-7cf29076760d', 'bae5c67a-7544-47fc-82ab-1f64a15acc62', '2777c4f8-b38b-4055-bed8-6f8a6121687a', 'e3d25322-95fe-4f72-9ee6-8ed6097d1b20', '094cdf61-c755-4e0d-bf23-bccfe8a9b976', 'dfd154de-b5c5-463c-9191-3632fd62a757', '41c08650-c4c7-4b9b-acd0-f7db8eaa717e', '32781ab6-af42-4e05-b3f0-da75c07bee16', '3e8cb307-97e0-44b3-9c85-97e9793b6cc4', '79cff0e3-4fe4-4a81-a58f-562fc43f2230', '84404282-ba53-47ce-bbda-8d62a02d8ddc', 'af7dc4b1-0f49-4e44-ad7b-14dee88302ee', 'b96613b8-47fc-4c55-a31b-83eb4b5b53d3', 'd25fcaf3-a466-4361-986b-aa6e4d626b24', 'd4ad6765-45bf-4ba5-9b57-e69eafb84293', '028f0591-3dfc-4a23-8a34-7dbabf7878ac', '9090c00a-cf96-496c-a923-9a97a4db68c0', 'a20439b7-3f57-4f32-b733-573e217401b1', '3d038e99-3de9-40db-adfe-c8110effdf15', '1eafeacb-1c22-4841-8349-8db5f89c5c91', 'ffc8f64a-ebbb-436b-a35e-b4d2551bc76a', 'e27e3def-371b-496c-8da4-6da680b3e9aa', '3d53887b-7c20-4d95-a721-e990b27204f0', '45614a51-9150-4b29-add2-c931e87b687d', '48a1dfc7-810e-46c2-b82c-0497471557fe', '26103bff-524f-4b5b-a063-c5e0a6ab55f5', '78b94735-b9ff-479f-bbfb-62b1ebfcdd1e', '0bf14e71-7e47-438b-9159-5b89d0705249', 'd7d78b1f-321f-4610-b013-14e77a54e69a', 'ef48a404-caf6-4abf-9737-0bb82f5a9a31', '00e9dfbb-d816-433b-a58b-5aed9358a3ea', '089d1139-3fcc-4d7a-83f5-469dd221d7f1', '3228d1ba-639f-4cef-8af5-8282efe9f504', 'a57a3d7d-94b1-4c2c-86e7-e63f92aa47da', '68602fb1-f79d-4369-a52a-247edb4169b6', 'ff7cfd49-6e1e-4ef6-bcbf-bd44be169295', '1282c2dc-f149-411b-8979-65cf2037550d', '5c9a046b-2a49-4488-b95d-992470bdc3a9', '6246e76e-c89b-4660-a1bc-b590908cc0ab', '7702c06d-6181-48d9-8aff-03d1bdcdeca1', 'c290cc60-48c5-4b96-b541-84a259ff3a65', 'c98f7946-5b0f-4f49-854a-a2180d36b717', 'e51f976c-1ab5-4f5f-98f4-c4d60c60e224', 'f6735a22-9d36-420a-a287-579cf6325266', 'fb8d128f-1c98-44fa-a947-8ec768916740', 'dc120dbe-5eac-4c9b-bedd-a44c8c2536a2', '46f456b1-1d6a-4b3a-874c-6b7fa4c19d67', '6638158c-1f83-482b-9c34-0c9b8aae911b', 'a9f2f3b0-78f7-48a5-b4ac-e1bcb671b9de', 'fbf617a1-0e7c-4241-986f-445f7a141471', '567822c9-6589-4cd3-9b6e-b25fab30098f', 'e9e2fe0d-dc86-414d-afe3-d6e53f8eb0bf', '0173cddb-506c-495f-9aac-5731d8166688', '6ec46c16-602c-496c-a203-53e27474b455', '758805cc-37c3-4d80-8301-41e42e92f6ca', 'a8bf6187-0b3b-430a-a0c6-da802d28ad78', 'd0393382-97be-480e-949f-fd080087b09d', '1a00de7a-c59f-4531-b269-9796d92f52c7', '1f7cc29b-a87c-48d5-a4ab-48c43e619753', '44501eee-4930-4659-a978-2dfc29a9aa03', '7527e921-7c5b-4bd4-aa3e-b61b51e01888', 'af336c12-5e70-463a-84a5-1cd32b51d187', '9553af20-accc-48f3-a6ee-ded7932309d3', '9aeb2967-9e81-4f45-8239-f8c445eea2b5', '514996a1-1a12-4b67-b480-674b0d82699f', '86e2f394-5af8-4a18-a461-f63674d1bbba', '3aca057e-995b-42b0-b000-db4ac9e9a3d4', '48a7fe31-8fff-4f8f-87b7-b843e68370c9', '9cce8f79-2715-4c42-be1a-63753765d510', '6f5b6a20-9aab-45ea-9de1-57ce177aedd1', 'bb12c45b-b0d6-4ca1-a583-abdb92bd3dc8', 'c858beb0-9250-45a0-946d-4f21fc208787', '06d75995-f28b-491f-8cd1-b1fa9aba3428', 'dc45e1e0-9277-4b0d-b6f2-eab10b78d43b', '17f22108-1899-4aa6-8fd4-d41634312df0', '25bb57ca-1c25-4802-add8-9cdbb2996a41', '370fecb6-7be4-48c4-8b40-eb247c0be5ab', '778dc2db-57dd-4f92-8bed-fb74879b93ad', '8c4384d5-e322-4d0c-a65c-d93bd73a1187', 'b25a7ae1-1bf4-4b5a-8f8f-8e17c7993337', '046dc0a2-17fd-4e50-b1b0-ac55946869e2', 'a92c8d2b-4498-46d6-a571-1a4d843fe05d', 'e2910d04-f01e-4a86-82a8-c73c3e48052e', 'ef281b09-c857-4d5a-9c43-b1ff3ff37525', '8ffc027e-d943-4caa-8f95-39e91624c350', '4a628ddb-c15b-4ae8-9036-2782209871b2', '2b965c56-6270-4773-a9d2-e0b7460d5d6f', '17d8f338-9b68-4d73-8107-cf9e8dffbea0', 'e2ca53b8-f2c6-4688-847a-9f8c023cc32c', 'e6f47d19-44c2-4c71-80b4-d25e99b8cd37', '092b3af1-311b-42eb-b0f6-a0b5212f2fe2', '1f5662ca-2edf-49ff-841b-1e2ec1261a64', '88fc2227-c298-4384-824d-dca6b8392301', 'aeb14001-91c5-49b9-9f0c-9a434667a038', 'b214d443-782f-4253-b500-b744fb57d789', '2b04734e-f49d-44f1-83bb-8df1c14f3d29', '45fb4ac6-9cb4-4673-943d-a5dcca76bba7', '5c22aaf4-4d45-4b39-891e-e3c0fe463cfe', '96d439fe-c379-42c5-b0fc-483d96fb1a8f', 'b1f94e26-c8c1-4283-8546-52064c93da7a', '2eabc5b6-aa2b-40b9-b2d8-9b67b7a65313', '92df367a-9a7c-40d3-ae8a-97171027557b', 'b0c3947c-a207-46d8-aa1b-04dc71d4e800', 'ba00d2eb-579f-42b2-9ad1-3fc86676ca3f', '0338ce8e-d2c5-41e2-8222-88144ba2e856', 'e6481e76-59c9-448e-a904-47c63d0a0c05', '555404e3-71a2-4576-b05d-457205e7c4ee', 'f9833003-7d06-4f6b-8950-de29b7fb541c', '3c3987dd-1110-4772-9af0-c1ac86684e24', '3cf0aae3-fca2-4d71-be2f-169e7c6c6e57', 'd875284e-7c56-4f26-87f8-fc3c509d2f3a', '2f6dfeed-8e7e-4956-a76c-fe4e6403f84f', '76e3130c-8c60-45d1-93cb-de0c68a9456b', 'af2ecc60-92f9-48d6-8822-a53fd62951cc', '2ab8a63d-16ed-4f1b-9456-779b8429d549', '1185ab63-4125-42dd-a858-4970539b6bb5', '5f545e10-0fca-494f-8a28-faf26558b0c3', '6d3a71ff-6eff-46e9-8015-009c899adb2e', '7ab963fe-8d7b-4ff0-925b-deb22ab9ce09', '95b0bddf-5a77-47d2-88a3-e919a8dd096b', 'a2c15803-dc25-4e00-a40e-53453b3dd1a0', 'cf8dd265-dc2e-4f1d-af26-1f9a8e87455c', 'e68d15e5-d5ba-48d5-8358-20edb875a174', 'fec1838f-f01c-4dd2-afb5-0c9a48cb9729', '23c6c17e-15d8-4f76-96c7-c92649ee7578', 'c47c426f-a67c-4032-937f-b04b31ca339c', 'e320398c-5517-49b2-9a81-bb56dd38fe61', 'd9eb4741-dea0-48ad-81b4-6b08b1ae9e95', 'e77dec9f-642b-4bea-96c9-d1a87259e47a', '4a4f9cf6-47c6-4ad1-b008-4e59e8c0eb96', '1127f942-1212-4803-bcf5-f594efc2df04', '4e404e87-6df1-4a72-b38c-883c8da1471b', '64721995-4479-477f-9e45-261f4b222135', 'a439ac26-549b-43a1-84d7-8888ed21d5f7', '4596a4d3-5995-4a32-8f77-2e4b19a9bd1d', '7c5e3937-1ea5-4a56-8abd-e39836b2bb7c', 'ce3b6301-93a3-4d6c-9b3c-bc4536ac8b8d', '97865518-e9ae-42d4-8e78-23ac1e617d6f', 'f1669e02-83e9-49b4-bf0a-afa7b7984830', '639717bb-f340-48ac-8ea9-6b669213d67c', '2e66e679-73a4-4f51-8a5c-254f70699706', 'd8290d24-3b48-431f-85e8-e27734fb26fa', 'e5226a02-798c-40f4-a9a4-a5d233fb929f', 'da169995-2b4b-4c12-a44a-d7d6400f595a', 'bdb3bdf5-99a8-4e7c-976c-301783eaf0e6', '59ee190e-4065-4a8f-9dff-4ef7f9fadd92', 'f3372a3a-437a-4661-970a-36b66dbfa780', '84e93ff2-7ff9-4201-a389-71de887544c8', '91468473-98b2-4424-a8aa-b66ddf7aacf8', '9792149b-242a-4811-9454-edab76c35b3c', 'bc698860-4dd5-4298-b033-c35b4f6f2cb4', 'cd89a85d-760b-4c9f-b86c-29373cc6e96c', 'db0a0a99-25f6-4e28-a2b2-101109a89413', '37ff68e6-7b9b-4289-8ae2-44b2e7a5a6a6', 'd41df096-1c03-4f8d-a857-7b906e93aa56', '7524fa94-51ad-4563-8016-e409a8a66116', '20d1d647-88b1-4c1c-a663-cbe2d155ce5a', '35a01eaf-fa5b-44e3-bb24-bbff0bd045da', '3fc5f77e-826c-48c3-a377-b8b45d1a91f2', '480d10f9-e822-4129-a74d-def7ef909cce', '72f36c63-1115-4e5e-975e-d74541bfbeb3', 'ede029af-3387-47f2-8160-6e1c1b120864', 'ffe2e0c0-0990-49f9-b0a8-834a89e87d48', '0954de07-b776-4329-91e6-1cb46324136f', 'd5314366-c9cc-4c1a-b94a-70a7c7f46606', '28514cd5-08c2-4464-aa8a-d7b89ba1d12d', '97e2b0df-b611-4486-bbc3-62815dceb653', '13b2a0fe-41f9-4f2a-8dba-1cf0a8a250da', 'faa8fece-0d1a-4381-bb50-7cb2a04ddb99', '09a61b23-bd9e-4ee8-93ef-dd23f47517c8', '1c4a86cf-d159-4965-baf2-26659af45429', '8a982f4b-7dad-4554-a8a8-81ac7371bf97', '9c7b376f-beb9-48b9-940a-a72605fa20d8', '7c566512-6da7-4a13-8a05-7766cc49ee6a', '839030d0-54a2-4c96-8849-ff2ed4c91814', 'a4c19357-d612-4717-83b1-a7fe153b4196', 'ac134c4b-a660-4104-b7d0-f07b2960e432', 'bc7e5ef2-1844-4cfd-a0e0-d72094a7d5af', '94b06864-d5a0-4109-ada3-076167e094da', '2707c4f8-f323-4a31-bfcb-5750fd1e67c9', 'a5134a8d-d287-4dca-a628-e063e20c91f1', 'bfe454f3-fe1d-42ac-ae4e-8a5b671d80df', '045feb3f-e604-4b00-8bf4-14406b6c540c', '2b73d67f-3567-4b78-b83b-2eeebfb78d85', '3a414e49-f67f-4c5b-878c-2b3d0174d6cb', '4b752c60-30a4-4a5b-837a-74793a734f91', '67f17adf-ec65-488b-aa07-d3a0903c8237', 'add21b64-6362-49c7-a8fb-2b329e096d65', 'b0ac41bd-911f-49e3-990b-a210d9d40c05', 'e4248595-1f42-4524-942e-5ff7c23b607e', 'defcec85-cf01-4276-b18f-bad66ae75b03', '50f0d807-fec5-4019-b6fa-bc8f4cca9ea3', 'f52fefb1-d3b2-4695-b9e2-1bb364f1d39c', '0da95d26-926a-41d9-8177-16ed4d132558', 'd4f1e238-9d61-4b9d-87bb-0142e0649c49', 'e58253e1-3001-40f8-8146-dcab1041e37a', 'd5fbcb7b-e326-45d4-a0fd-aab6bf53f500', 'a9846286-2312-4926-8995-9727ec9e8f67', '2bd0b11a-1963-4727-9bc1-619fa01e6804', 'c849c72e-19b6-4f25-bd79-62b78a8a13f2', 'dfed45a5-0690-4b54-91cc-9db890ea898c', 'e9208b5e-48dc-41f6-a6a2-5a8a56e97c6c', 'f5ba694f-d760-4a09-ac92-853811e9ebb6', '101a7f88-3f94-4b0d-a4d4-2204ad2ddafa', '2fa116b6-da16-4302-bd00-0f413da5d24e', 'b0f9d6dc-f91a-4551-a6d2-fc85f74fbc73', '13ec8297-6f6a-46cd-b3d8-8c827e3e4a42', 'f102d6ee-2d9e-4aa0-b1f8-160e00727224', '7411a96b-f324-46ac-a607-9b57b0023e17', '4c61cd12-a7b1-4609-b6fd-43b39755c1f7', 'b84d0967-e2b2-46f6-9ad8-754f841f8c2b', 'cfff13ad-a7f3-4fa7-98a4-4d31b29d3b14', 'a7139a1b-9527-48b6-9585-4ec6e40bbe52', '2c582b5f-b5a2-47a1-9150-a278699efad5', '39506869-09f0-46cf-a50c-4d5ab1b40d51', '483585f4-a495-4e74-b7ca-27a5880df909', '6befa8b0-3d24-45bc-9f53-0dd89d070b6f', '6fa93fe5-4e80-4654-bf4c-c5dc53fe5dd5', '9bfbc7c5-5af1-4b23-b478-8453b053fce8', 'e7cd6d88-7397-40d0-aa61-33876854ec29', 'f2556bb8-8b8e-43c0-b960-5a00e434a52d', '2db837d9-9c67-487f-a18b-a04a7a348b71', 'e570b5c7-8d7e-4fff-9669-580df3f739dc', '4887a5d0-896d-47fa-9c0b-0247af55b7ae', '828d052b-3de5-4da9-ac29-d1682de1fbb0', 'dfa8519d-46d5-48af-a840-f712c5042aea', 'fe6f0ca1-9e2b-46c9-b52a-219138cb3170', '17ba8808-4ef7-4f28-9129-5a42f8f83e90', '08520138-46c0-451d-861a-41fc14cf65b6', '2d590b12-a3c8-4d85-9fe1-83ff9d83750f', '72dfbc00-46d6-4574-a8f9-be23b16df482', '84f7c68f-151b-4d68-a0fb-a87d6cbafcd6', '249ef90f-641c-4bf8-b8f8-a410c1fa591e', '0a8150ba-227b-4fa3-bc78-bcc5984d9abf', '13424f21-9677-4703-a80a-3e8f53fb3288', '1dd5c215-26f0-43c4-8a09-9336d7e886e6', '29fd0f9e-4877-4a5a-b6c8-f144d52405f4', '31eaf1fb-a4cf-4526-ab37-8d31b4fca07d', '428ce37c-f7e4-4a26-bfdb-dfb9c0bd42e0', '566b024c-0322-463d-8f5e-d249f779a460', 'b3142d4c-493b-4b40-bc80-d8d8d23eab74', '0e4f459a-03c2-4cf6-9ac4-fc88a1e31379', '7d197ab5-b31c-4fdd-8cd6-d1dcddd584c1', '9bbf9c30-31da-48af-891b-80dcb749af67', '1e5a5161-1985-4b4b-87a5-d949bdba76f2', '1ebae8d4-7efb-46d5-b06a-d9d1daa07a3b', '37190206-d9af-4e58-a9aa-ee9b9f850853', '4a2e47be-8ead-4e95-80e4-7643ae147ac8', 'a32e0afe-3577-4966-accc-064827164571', '2fb5c7c3-4f00-4e10-a45d-8d1fdca8aaf4', '4bd07b79-1481-4d09-a793-b96a1a18eac2', '53d070a5-3bdd-4010-9a1c-cc4861f27a7b', 'c6bbdfad-65a9-41d4-a4a3-66bb9d991a0e', 'f53e9828-4ff5-4d34-9644-8fdb3f469709', '09a1bd15-6aca-462b-8fc9-a46e0626f028', '2ddbd32c-0d59-473c-a42d-677a1173a9bb', '4dd19385-37ce-4aef-9d11-6422dc5d884c', '9ba16a8e-2548-4380-ac95-387b1759ccb1', 'd62745f5-53e3-422d-8c38-ad2a83a3b9be', '9125c17a-445b-4c8a-831e-57468f0f09a6', 'fdeca262-86fe-41db-bf8e-5384e6ac0c3c', '0880b4cc-0f06-451c-848a-20f3b103307a', '42cd2a3b-4b24-4bf5-8523-746d22182d2b', '59108572-1982-4bb3-985b-50f6745251d4', '80bc0e50-d3b3-4c3a-bb18-f2999a25a421', 'c14778f8-047f-4161-8e02-946ea7eda937', 'd8fd0126-fa29-447c-801c-cbe0939a450f', 'e9e2e0d6-9f8f-4493-9ab3-009eb9aa0621', '3ed2521f-a18e-4eaf-b31c-5dafe2130e07', 'd9cf21f1-6e7c-4cc3-a91c-e1337e0458f2', '5b93c4e4-c18e-4e44-9f32-aa58e743ea88', '89d60d6e-4012-49f4-a129-0c986b0d20ce', 'fe84b3d0-d22c-4ed8-afb0-55c0fb3218bd', '137fa172-16d7-4403-99a7-7fae524ad93d', '917f876c-e280-4e0f-a9fe-11871d09c332', 'ee0423ff-c782-469b-bbf5-7cb44c66a81e', '3108b0d2-fd65-4f1d-83c0-215c40dab3a7', '1bb00179-db42-45f8-94b0-9cef05a72753', '8704922d-33a2-438c-9ed3-6ca72b42c5fd', 'b330bd42-7325-4e42-8c73-6ec05e9d5d5c', 'e87ff15d-4395-45f9-be1f-af8e85e60039', '0c1cfb26-2d1c-4fe5-b1d0-7320af4a63f1', '7b7b562e-6f30-47ab-accb-d5fe49ca1bcd', '1ea49207-33e9-4ab9-8c0e-4636e0deccc4', '3394a5f6-df52-40f4-809a-e613945e25ad', '3e1f0bd3-5de0-4ee5-b8b9-d7e25702aaef', '5aa58e64-594f-467a-91c4-1f2d3e6218dc', '637971b1-21d8-4098-8889-4a22c473d145', '7b874cad-88fb-4c54-971c-c992c247d382', '92e19e72-8c3d-44f4-b0e7-e2f9273f1ab2', 'cd63a6ca-8824-4914-849e-450256e0036e', 'cdfaaa6a-1605-4fbc-86c8-79154b46c43f'], '@persistence_version': 1} 2023-01-24T08:07:58.566 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527868758):Number of requests made: 1; Number of events received: 2000; Number of duplicated events filtered out: 1683; Number of events generated and sent: 317; Average of events per second: 32.369. 2023-01-24T08:07:58.566 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> The data is up to date! 2023-01-24T08:07:58.566 INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Data collection completed. Elapsed time: 9.808 seconds. Waiting for 590.192 second(s) until the next one
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Statistics for this pull cycle (@devo_pulling_id=1674492783536):Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000. 2023-01-23T22:23:05.265 INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> The data is up to date!
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
historical_date_utc
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error Type | Error Id | Error Message | Cause | Solution |
InitVariableError | 1 |
| historic_date_utc is mentioned in the wrong date time format. | Write the historic_date_utc in correct formatFor. ex. |
InitVariableError | 2 |
| historic_date_utc is mentioned in the wrong date time format. | Write the historic_date_utc in correct formatFor. ex. |
InitVariableError | 3 |
| The historic datetime mentioned is of the future. | The value of historic datetime should always be lesser than current datetime |
SetupError | 100 |
| Not able to establish connection/ pull data due to incorrect credentials or account expired. | Re-check if the entered credentials are correct, and that the access token is expired or not |
PullError | 300 |
| Invalid token or request | Re-check if the entered credentials are correct, and that the access token is expired or not |
PullError | 301 |
| Cause for this error will be you have entered invalid values in service parameters. | Check whether the parameters for that service are correct or not. |
PullError | 302 |
| Cause for this error will be you have entered invalid values for client_id, client_secret or account_id | Re-enter correct values for
|
PullError | 303 |
| Incorrect Url or endpoint | Re-enter |
PullError | 304 |
| Enter invalid value in config.yaml file for that service. | Check whether the parameters for that service are correct or not. |
PullError | 305 |
| Cause for this error will be written in the Error message on the console. | As there are a number of causes for this error, check on the console as the reason is described and take action accordingly. |
Collector operations
This section is intended to explain how to proceed with the specific operations of this collector.
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
INFO MainProcess::MainThread -> Added "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula" directory to the Python path 2023-02-22T10:11:12.878 INFO MainProcess::MainThread -> Added "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula/config_internal" directory to the Python path 2023-02-22T10:11:12.878 INFO MainProcess::MainThread -> Added "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula/schemas" directory to the Python path 2023-02-22T10:11:12.878 INFO MainProcess::MainThread -> Production mode: False, execute only setup and exit: False, Python version: "3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]", current dir: "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula", exists "config" dir: True, exists "config_internal" dir: True, exists "certs" dir: True, exists "schemas" dir: True, exists "credentials" dir: True 2023-02-22T10:11:12.900 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config.yaml", "job_config_loc": null, "collector_config_loc": null} 2023-02-22T10:11:12.900 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json" 2023-02-22T10:11:12.901 INFO MainProcess::MainThread -> "/etc/devo/job" does not exists 2023-02-22T10:11:12.901 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json" 2023-02-22T10:11:12.901 INFO MainProcess::MainThread -> "/etc/devo/collector" does not exists 2023-02-22T10:11:12.901 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula/config/config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False} 2023-02-22T10:11:12.917 INFO MainProcess::MainThread -> Build time: "UNKNOWN", OS: "Linux-5.15.0-60-generic-x86_64-with-glibc2.29", collector(name:version): "example_collector:1.0.0", owner: "integrations_factory@devo.com", started at: "2023-02-22T04:41:12.909599Z" 2023-02-22T10:11:12.919 INFO MainProcess::MainThread -> Initialized all object from "MainProcess" process 2023-02-22T10:11:12.919 INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=120s) 2023-02-22T10:11:12.921 INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=120s) 2023-02-22T10:11:12.921 INFO OutputProcess::MainThread -> Process started 2023-02-22T10:11:12.924 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Number of available senders: 1, sender manager internal queue size: 0 2023-02-22T14:26:39.858 INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> enqueued_elapsed_times_in_seconds_stats: {} 2023-02-22T14:26:39.858 INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Sender: DevoSender(standard_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False} 2023-02-22T14:26:39.858 INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Standard - Total number of messages sent: 31, messages sent since "2023-02-22 08:51:39.785516+00:00": 31 (elapsed 0.009 seconds) 2023-02-22T14:26:39.864 INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> Number of available senders: 1, sender manager internal queue size: 0 2023-02-22T14:26:39.865 INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> enqueued_elapsed_times_in_seconds_stats: {} 2023-02-22T14:26:39.865 INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> Sender: ConsoleSender(standard_senders,console_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} 2023-02-22T14:26:39.865 INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> Standard - Total number of messages sent: 31, messages sent since "2023-02-22 08:51:39.798224+00:00": 31 (elapsed 0.006 seconds) 2023-02-22T14:26:39.876 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Number of available senders: 1, sender manager internal queue size: 0 2023-02-22T14:26:39.876 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> enqueued_elapsed_times_in_seconds_stats: {} 2023-02-22T14:26:39.876 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False} 2023-02-22T14:26:39.876 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Lookup - Total number of messages sent: 0, messages sent since "2023-02-22 08:51:39.809055+00:00": 0 (elapsed 0.000 seconds) 2023-02-22T14:26:39.879 INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> Number of available senders: 1, sender manager internal queue size: 0 2023-02-22T14:26:39.879 INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> enqueued_elapsed_times_in_seconds_stats: {} 2023-02-22T14:26:39.880 INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> Sender: ConsoleSender(lookup_senders,console_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} 2023-02-22T14:26:39.880 INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> Lookup - Total number of messages sent: 0, messages sent since "2023-02-22 08:51:39.820290+00:00": 0 (elapsed 0.000 seconds) 2023-02-22T14:26:39.883 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Number of available senders: 1, sender manager internal queue size: 0 2023-02-22T14:26:39.883 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> enqueued_elapsed_times_in_seconds_stats: {} 2023-02-22T14:26:39.883 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Sender: DevoSender(internal_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} 2023-02-22T14:26:39.884 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Internal - Total number of messages sent: 59, messages sent since "2023-02-22 08:51:39.830694+00:00": 59 (elapsed 0.578 seconds) 2023-02-22T14:26:39.884 INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> Number of available senders: 1, sender manager internal queue size: 0 2023-02-22T14:26:39.884 INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> enqueued_elapsed_times_in_seconds_stats: {} 2023-02-22T14:26:39.884 INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> Sender: ConsoleSender(internal_senders,console_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} 2023-02-22T14:26:39.884 INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> Internal - Total number of messages sent: 59, messages sent since "2023-02-22 08:51:39.841277+00:00": 59 (elapsed 0.007 seconds) 2023-02-22T14:26:44.862 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Pull Started 2023-02-22T14:26:44.862 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> Consumed messages: 5, total_bytes: 3778 (60.00074 seconds) 2023-02-22T14:26:44.863 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:44.862", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"Pull Started\", \"time\": \"2023-02-22T08:56:44.862220Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"} 2023-02-22T14:26:44.863 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> Consumed messages: 5 messages (60.000787 seconds) => 0 msg/sec 2023-02-22T14:26:44.863 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 5 messages (60.000704 seconds) => 0 msg/sec 2023-02-22T14:26:48.548 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections 2023-02-22T14:26:48.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Total detections available are = 31 2023-02-22T14:26:48.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Received 31 detections data from Nebula Server 2023-02-22T14:26:48.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detections if present... 2023-02-22T14:26:48.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Saved state: {'last_polled_timestamp': 1677056144.861205, 'historic_date_utc': None, 'ids_with_same_timestamp': ['0ab0b92f-4653-5f9a-870a-4c4aa128072d'], '@persistence_version': 1} 2023-02-22T14:26:48.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Received 0 detections data from Nebula Server after removing duplicates 2023-02-22T14:26:48.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Number of detections sent to Devo: 0 2023-02-22T14:26:48.549 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> State last_polled_timestamp is updated with retrieving timestamp 2023-02-22T14:26:48.550 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000. 2023-02-22T14:26:48.550 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000. 2023-02-22T14:26:48.550 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> The data is up to date! 2023-02-22T14:26:48.551 INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Data collection completed. Elapsed time: 3.689 seconds. Waiting for 56.311 second(s) until the next one 2023-02-22T14:26:48.553 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.550", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"(Partial) Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.\", \"time\": \"2023-02-22T08:56:48.550458Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"} 2023-02-22T14:26:48.553 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.550", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.\", \"time\": \"2023-02-22T08:56:48.550726Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"} 2023-02-22T14:26:48.553 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.550", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"The data is up to date!\", \"time\": \"2023-02-22T08:56:48.550900Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"} 2023-02-22T14:26:48.553 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.551", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"Data collection completed. Elapsed time: 3.689 seconds. Waiting for 56.311 second(s) until the next one\", \"time\": \"2023-02-22T08:56:48.551084Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"} 2023-02-22T14:27:39.966 INFO InputProcess::MainThread -> [GC] global: 32.8% -> 32.8%, process: RSS(45.85MiB -> 45.85MiB), VMS(503.38MiB -> 503.38MiB) 2023-02-22T14:27:39.967 INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:57:39.966", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"[GC] global: 32.8% -> 32.8%, process: RSS(45.85MiB -> 45.85MiB), VMS(503.38MiB -> 503.38MiB)\", \"time\": \"2023-02-22T08:57:39.966130Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null}"} 2023-02-22T14:27:39.967 INFO InputProcess::MainThread -> global_status: {"input_process": {"process_id": 3653, "process_status": "running", "thread_counter": 7, "thread_names": ["MainThread", "QueueFeederThread", "NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined)", "QueueFeederThread", "NebulaDetectionDataPuller(example_input,12345,detections,predefined)", "ServiceThread(example_input,12345,detections,predefined)", "InputThread(example_input,12345)"], "memory_info": {"rss": "45.85MiB", "vms": "503.38MiB", "shared": "9.52MiB", "text": "2.42MiB", "lib": "0.00B", "data": "84.34MiB", "dirty": "0.00B"}, "input_threads": [[]], "running_flag": true, "message_queues": {"standard": {"name": "standard_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7fdc9e4b3df0>"}, "lookup": {"name": "lookup_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7fdc9e4e1f10>"}, "internal": {"name": "internal_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7fdc9e465700>"}}}} 2023-02-22T14:27:40.001 INFO OutputProcess::MainThread -> [GC] global: 32.8% -> 32.8%, process: RSS(43.34MiB -> 43.34MiB), VMS(1.55GiB -> 1.55GiB)
By default, these information traces will be displayed every 10 minutes.
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal
, standard
, and lookup
). This collector uses the following Sender Services:
Sender services | Description
|
| Displays the number of concurrent senders available for the given Sender Service. |
| Displays the items available in the internal sender queue. This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
| Displays the number of events from the last time and following the given example, the following conclusions can be obtained:
By default, these traces will be shown every 10 minutes. |
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace | Description |
---|---|
| Displays the number of concurrent senders available for the given Sender Service |
| Displays the items available in the internal sender queue. |
| Displays the number of events from the last time and following the given example, the following conclusions can be obtained:
|
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the
global
value.All metrics (Global, RSS, VMS) include the value before freeing and after
previous -> after freeing memory
INFO InputProcess::MainThread -> [GC] global: 31.1% -> 31.2%, process: RSS(39.55MiB -> 39.77MiB), VMS(493.59MiB -> 493.68MiB) 2023-02-22T10:46:23.332 INFO OutputProcess::MainThread -> [GC] global: 31.1% -> 31.2%, process: RSS(39.11MiB -> 40.75MiB), VMS(1.54GiB -> 1.54GiB)
Differences between RSS
and VMS
memory usage:
RSS
is the Resident Set Size, which is the actual physical memory the process is usingVMS
is the Virtual Memory Size which is the virtual memory that process is using
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.
Change log for v1.x.x
Release | Released on | Release type | Details | Recommendations |
---|---|---|---|---|
v1.0.0 | 12 May 2023 | New collector | - | - |
- No labels