Page Comparison

Table of Contents

minLevel	2
maxLevel	2
type	flat

Overview

Malwarebytes Nebulais a cloud-hosted security operations platform that allows you to manage control of any malware or ransomware incident

Devo collector features

Feature	Details
Allow parallel downloading (`multipod`)	`Not allowed`
Running environments	`Collector server` `On-premise`
Populated Devo events	`Table`
Flattening preprocessing	`No`

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

Info

Configuration requierementsrequirements

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details
`client_id`	Credential client ID.
`client_secret`	Credential client secret.
`account_id`	Credential account ID.
`api_base_url`	Credential API base url.

Info
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Overview

Malwarebytes Nebula is a cloud-hosted security operations platform that allows you to manage control of any malware or ransomware incident

Devo collector features

...

Feature

...

Details

...

Allow parallel downloading (multipod)

...

Not allowed

Allowed

...

Running environments

...

Collector server

On-premise

...

Populated Devo events

...

Table

...

Flattening preprocessing

...

Data sources

Data Source	Description	API Endpoint	Collector service name	Devo Tabletable	Available from release
Notifications	Malwarebytes Nebula can notify you when certain events occur, such as when real-time protection or scheduled scans detect threats, or if a new endpoint registers to your console.	`<base_url>/notifications/subscriptions`	notifications	`myedr.appmalwarebytes.nebula.notificationsnotification`	`v1.0.0`
Detection	The Detections section in Malwarebytes Nebula displays information on all threats, and potential threats, with the action taken for each item found on endpoints in your environment	`<base_url>/detections`	detections	`myedr.appmalwarebytes.nebula.detectionsdetection`	`v1.0.0`
Events	Event is a general term for a threat that has occurred, remediation or other action taken on a threat, and other endpoint-related activity.	`<base_url>/events`	events	`myedr.appmalwarebytes.nebula.eventsevent`	`v1.0.0`
Vulnerability Management	shows vulnerabilities for installed software and operating systems on managed endpoints.	`<base_url>/cve/export` `<base_url>/cve/{id}`	vulnerability_management	`myedr.appmalwarebytes.nebula.vulnerabilitymanagementvulnerability`	`v1.0.0`
Suspicious activity	Suspicious Activity Monitoring is a feature included in Malwarebytes Endpoint Detection and Response	`<base_url>/sa`	suspicious_activity	`myedr.appmalwarebytes.nebula.suspiciousactivitysuspicious_activity`	`v1.0.0`
DNS Logs Data	Logs of Dns data	`<base_url>/dns`	dns_log_data	`myedr.appmalwarebytes.nebula.dnslogdatadns_logdata`	`v1.0.0`

For more information on how the events are parsed, visit our page.

Vendor setup

There are some steps you need to follow to run the collector.

Rw ui steps macro

Rw step

Use your email and password to login into Malware Nebula Cloud.

Image Removed

Image Added

Rw step

Go to Settings → API & Integrations → Add.

Rw step

Provide necessary access to the credentials.

Note

Credential only shows once

Please, note down your credentials as they only show once.

Image Removed

Image Added

Image Removed

Image Added

Rw step

A new web client will be created.

Image Removed

Image Added

Anchor

	Accepted-authentication-methods
	Accepted-authentication-methods

Accepted authentication methods

Authentication Method

Username

Password

client_id/client_secret

Status

colour	Green
title	Required

Status

colour	Green
title	Required

account_id

Status

colour	Green
title	Required

Status

colour	Green
title	Required

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Rw ui tabs macro

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Image Removed

Image Added

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

globals:
  debug: false
  id: not used
  name: example_collector
  persistence:
    type: filesystem
    config:
      directory_name: state

outputs:
  devo_eu_1:
    type: devo_platform
    config:
      address: collector-eu.devo.io
      port: 443
      type: SSL
      chain: chain.crt
      cert: if_metronlabs.crt
      key: if_metronlabs.key
 
 console_1:
   type: console

inputs:
  example_input:
    id: 12345
    enabled: true
    requests_per_second: 5
    api_base_url: https://api.malwarebytes.com
    credentials:
      client_id:
      client_secret:
      account_id:
    services:
      events:
        request_period_in_seconds: 600
        historic_date_utc:               #example date = 2022-11-15T14:32:33Z
        override_tag: myedr.appmalwarebytes.nebula.eventsevent
     
      notifications:
        request_period_in_seconds: 10
        historic_date_utc:        #example date = 2022-11-23T08:12:57.422Z
        override_tag: myedr.appmalwarebytes.nebula.notificationsnotification

      suspicious_activity:
        request_period_in_seconds: 600
        historic_date_utc:          #example date = 2022-11-23T08:12:57.422Z
        override_tag: myedr.appmalwarebytes.nebula.suspiciousactivitysuspicious_activity

      vulnerability_management:
        request_period_in_seconds: 600
        historic_date_utc:             #example date = 2022-11-15T14:32:33Z
        override_tag: myedr.appmalwarebytes.nebula.vulnerabilitymanagementvulnerability
       
         detections:
        request_period_in_seconds: 600
        historic_date_utc:                #example date = 2022-11-23T08:12:57.422Z
        override_page_size: 2000
        override_tag: myedr.appmalwarebytes.nebula.detectionsdetection

      dns_log_data:
        request_period_in_seconds: 600
        historic_date_utc:                #example date = 2022-11-23T08:12:57.422Z
        override_page_size: 2000
        override_tag: myedr.appmalwarebytes.nebula.dnslogdatadns_logdata

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Replace the placeholders with your required values following the description table below:

Parameter	Data type	Type	Value Range	Details
`debug_status`	`bool`	`Mandatory`	`false` / `true`	If the value is `true`, the debug logging traces will be enabled when running the collector. If the value is `false`, only the `info`, `warning` and `error` logging levels will be printed.
`collector_id`	`int`	`Mandatory`	Minimum length: 1 Maximum length: 5	Use this param to give an unique id to this collector.
`collector_name`	`str`	`Mandatory`	Minimum length: 1 Maximum length: 10	Use this param to give a valid name to this collector.
`devo_address`	`str`	`Mandatory`	`collector-us.devo.io` `collector-eu.devo.io`	Use this param to identify the Devo Cloud where the events will be sent.
`chain_filename`	`str`	`Mandatory`	Minimum length: 4 Maximum length: 20	Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: `chain.crt`
`cert_filename`	`str`	`Mandatory`	Minimum length: 4 Maximum length: 20	Use this param to identify the `file.cert` downloaded from your Devo domain.
`key_filename`	`str`	`Mandatory`	Minimum length: 4 Maximum length: 20	Use this param to identify the `file.key` downloaded from your Devo domain.
`input_id`	`int`	`Mandatory`	Minimum length: 1 Maximum length: 5	Use this param to give an unique id to this input service. This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
`input_status`	`bool`	`Mandatory`	`false` / `true`	Use this param to enable or disable the given input logic when running the collector. If the value is `true`, the input will be run. If the value is `false`, it will be ignored.
`api_base_url`	`str`	`Mandatory`		Use this param to define the URL used by the collector to pull data.
`client_id`	`str`	`Mandatory`	Minimum length: 1	client_id of the Nebula server
`client_secret`	`str`	`Mandatory`	Minimum length: 1	client_secret of the Nebula server
`account_id`	`str`	`Mandatory`	Minimum length: 1	account_id of the Nebula server
`historic_date_utc`	`str`	`Optional`	UTC with format(use examples given for each service)	This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (1 month back for example) before downloading new events.
`request_period_in_seconds_value`	`int`	`Optional`	Minimum length: 1	Period in seconds used between each data pulling, this value will overwrite the default value (600 seconds)

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-malwarebytes_nebula_if-docker-image-1.0.0	`7e450b1ed6bda604a2d52e57e3ecc8d4e2e312aedaee6e5f9217ecff8713533b`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Events service

Expand

title	Internal process and deduplication method

Data is first pulled for events based on the historic date provided or default historic days, event_id of the last event is stored in a state file, and data is sorted manually in descending order, the last event will be the old one. New data is compared with the previously stored event id to identify the duplicate items and removed them.

Expand

title	Devo categorization and destination

All events of Events service are ingested into the table myedr.appmalwarebytes.nebula.eventsevent

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

INFO InputProcess::MainThread -> NebulaEventsDataPuller(example_input,12345,events,predefined) - Starting thread
2023-01-23T16:16:31.386 WARNING InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Waiting until setup will be executed
2023-01-23T16:16:31.386    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Token has expired. Generating the new one
2023-01-23T16:16:31.387 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> The token/header/authentication is expired and it needs to be refreshed
2023-01-23T16:16:31.388    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Requesting access token from the Nebula server
2023-01-23T16:16:31.402    INFO OutputProcess::MainThread -> [GC] global: 25.0% -> 25.0%, process: RSS(46.83MiB -> 47.60MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-23T16:16:31.408    INFO InputProcess::MainThread -> [GC] global: 25.0% -> 25.0%, process: RSS(46.96MiB -> 47.29MiB), VMS(791.23MiB -> 791.48MiB)
2023-01-23T16:16:31.720    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140563744962544"
2023-01-23T16:16:31.721    INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140563744962400"
2023-01-23T16:16:32.343    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Requesting access token from the Nebula server
2023-01-23T16:16:32.344    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 16:46:31
2023-01-23T16:16:32.344    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 
2023-01-23T16:16:32.344    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Setup for module <NebulaEventsDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

023-01-24T08:03:26.575    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Pull Started
2023-01-24T08:03:27.586    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/events?start=2023-01-24T02:32:26Z
2023-01-24T08:03:27.588    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Removing the duplicate events if present...
2023-01-24T08:03:27.589    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Number of events sent to Devo: 0
2023-01-24T08:03:27.589    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Total number of events: 0
2023-01-24T08:03:27.590    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-24T08:03:27.591    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Saved state: {'last_polled_timestamp': 1674527606.575356, 'historic_date_utc': None, 'ids_with_same_timestamp': ['0fa33de2-963a-4b7f-b709-4111eb82712c'], '@persistence_version': 1}
2023-01-24T08:03:27.591    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527606575):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-01-24T08:03:27.593    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> The data is up to date!
2023-01-24T08:03:27.595    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Data collection completed. Elapsed time: 1.019 seconds. Waiting for 58.980 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

2023-01-24T08:03:27.591    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527606575):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-01-24T08:03:27.593    INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> The data is up to date!

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Expand

title	Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_date_utc parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

Vulnerability management service

Expand

title	Internal process and deduplication method

Data is first polled for vulnerability_id based on the historic data provided or default historic days, vulnerability_id of the last vulnerability is stored in a state file, and data is sorted manually in descending order, the last vulnerability will be the old one. New data is compared with the previously stored vulnerability_id to identify the duplicate items and removed them.
Based on each vulnerability_id a description of the vulnerability is obtained in the next API call, after removing duplicates for ids.

Expand

title	Devo categorization and destination

All events of Vulnerability service are ingested into the table myedr.appmalwarebytes.nebula.vulnerability_management.

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s)
2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> ServiceThread(example_input,12345,vulnerability_management,predefined) - Starting thread (execution_period=60s)
2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Starting thread
2023-01-23T17:09:18.003    INFO InputProcess::MainThread -> NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) - Starting thread
2023-01-23T17:09:18.003    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Token has expired. Generating the new one
2023-01-23T17:09:18.004 WARNING InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Waiting until setup will be executed
2023-01-23T17:09:18.004 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> The token/header/authentication is expired and it needs to be refreshed
2023-01-23T17:09:18.005    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server
2023-01-23T17:09:18.020    INFO OutputProcess::MainThread -> [GC] global: 25.8% -> 25.9%, process: RSS(46.42MiB -> 48.71MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-23T17:09:18.029    INFO InputProcess::MainThread -> [GC] global: 25.9% -> 25.9%, process: RSS(47.31MiB -> 47.38MiB), VMS(791.48MiB -> 791.48MiB)
2023-01-23T17:09:18.341    INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332628086400"
2023-01-23T17:09:18.344    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332642608512"
2023-01-23T17:09:19.010    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server
2023-01-23T17:09:19.011    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 17:39:18
2023-01-23T17:09:19.012    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 
2023-01-23T17:09:19.012    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Setup for module <NebulaVulnerabilityDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

2023-01-23T17:19:40.513    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Pull Started
2023-01-23T17:19:41.573    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/export
2023-01-23T17:19:41.574    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Received 5 CVE ids from Nebula Server
2023-01-23T17:19:41.575    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Removing the duplicate cve if present...
2023-01-23T17:19:41.575    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2022-34716'}
2023-01-23T17:19:42.498    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2022-34716
2023-01-23T17:19:42.499    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2022-24464'}
2023-01-23T17:19:43.419    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2022-24464
2023-01-23T17:19:43.419    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2020-8927'}
2023-01-23T17:19:44.393    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2020-8927
2023-01-23T17:19:44.395    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2021-34485'}
2023-01-23T17:19:45.339    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2021-34485
2023-01-23T17:19:45.341    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2021-26423'}
2023-01-23T17:19:46.356    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2021-26423
2023-01-23T17:19:46.359    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Number of vulnerabilities sent to Devo: 5
2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Saved state: {'last_polled_timestamp': 1674474580.484891, 'historic_date_utc': 1669991553.0, 'ids_with_same_timestamp': ['CVE-2021-26423'], '@persistence_version': 1}
2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674474580484):Number of requests made: 1; Number of events received: 5; Number of duplicated events filtered out: 0; Number of events generated and sent: 5; Average of events per second: 0.855.
2023-01-23T17:19:46.362    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> The data is up to date!
2023-01-23T17:19:46.363    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Data collection completed. Elapsed time: 5.879 seconds. Waiting for 594.121 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674474580484):Number of requests made: 1; Number of events received: 5; Number of duplicated events filtered out: 0; Number of events generated and sent: 5; Average of events per second: 0.855.
2023-01-23T17:19:46.362    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> The data is up to date!

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Expand

title	Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_date_utc parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

Notifications service

Expand

title	Internal process and deduplication method

All of the notifications will be fetched from the API call, then based on the historic date provided or default historic days we will return notifications with a timestamp higher than the most recent timestamp for that Notification, we will update the state with the id’s and its log time which are polled after removing duplicates.
Hence in the next pull the ids present in the state file will not be pulled again and duplicates will be removed.

Expand

title	Devo categorization and destination

All notifications of Notifications service are ingested into the table myedr.appmalwarebytes.nebula.notificationsnotification

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s)
2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> ServiceThread(example_input,12345,vulnerability_management,predefined) - Starting thread (execution_period=60s)
2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Starting thread
2023-01-23T17:09:18.003    INFO InputProcess::MainThread -> NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) - Starting thread
2023-01-23T17:09:18.003    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Token has expired. Generating the new one
2023-01-23T17:09:18.004 WARNING InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Waiting until setup will be executed
2023-01-23T17:09:18.004 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> The token/header/authentication is expired and it needs to be refreshed
2023-01-23T17:09:18.005    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server
2023-01-23T17:09:18.020    INFO OutputProcess::MainThread -> [GC] global: 25.8% -> 25.9%, process: RSS(46.42MiB -> 48.71MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-23T17:09:18.029    INFO InputProcess::MainThread -> [GC] global: 25.9% -> 25.9%, process: RSS(47.31MiB -> 47.38MiB), VMS(791.48MiB -> 791.48MiB)
2023-01-23T17:09:18.341    INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332628086400"
2023-01-23T17:09:18.344    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332642608512"
2023-01-23T17:09:19.010    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server
2023-01-23T17:09:19.011    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 17:39:18
2023-01-23T17:09:19.012    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 
2023-01-23T17:09:19.012    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Setup for module <NebulaVulnerabilityDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

2023-01-23T17:45:13.730    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Pull Started
2023-01-23T17:45:15.115    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/notifications/subscriptions
2023-01-23T17:45:15.116    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Removing the duplicate events if present...
2023-01-23T17:45:15.117    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Number of notifications sent to Devo: 12
2023-01-23T17:45:15.117    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Total number of notifications: 12
2023-01-23T17:45:15.118    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-23T17:45:15.118    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Saved state: {'last_polled_timestamp': 1674476113.707944, 'historic_date_utc': 1666512777.422, 'ids_with_same_timestamp': {'99316cff-92c6-45ee-881b-bb95b18fd1f5': '2022-12-19T12:11:19.183Z', '62ae6d64-cfce-4da2-9617-aea52d64eabb': '2022-12-19T11:55:28.794Z', 'eeec1ead-8ae3-4303-a774-10c5f33292be': '2022-12-16T11:03:09.881Z', '1ce022f6-543f-4f3a-a14e-e91f37360c53': '2022-12-15T07:39:42.715Z', 'b175c54b-5631-4988-b718-48cd5b06d933': '2022-12-15T06:49:24.427Z', '763e40d3-b33b-4872-b8f1-10a08e169989': '2022-12-15T06:34:16.769Z', 'd972ff54-bde5-4512-b354-46f21894bdd6': '2022-12-14T11:13:08.881Z', '8d0db3ba-5466-4c5d-9ad3-3530eb878596': '2022-12-14T08:42:56.454Z', '166106ff-fa40-4607-b12d-44aef22501c2': '2022-12-14T08:41:19.339Z', 'b37b5dbc-b4e9-4392-aa96-de105dd24ad7': '2022-12-06T06:11:36.628Z', '0ad46e4e-df07-438a-8996-a0e08672e926': '2022-12-02T03:48:34.896Z', '7ee4c548-9f31-4439-a65d-91e31b713ad4': '2022-11-23T08:12:57.422Z'}, '@persistence_version': 1}
2023-01-23T17:45:15.118    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674476113707):Number of requests made: 1; Number of events received: 12; Number of duplicated events filtered out: 0; Number of events generated and sent: 12; Average of events per second: 8.646.
2023-01-23T17:45:15.118    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> The data is up to date!
2023-01-23T17:45:15.119    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> Data collection completed. Elapsed time: 1.411 seconds. Waiting for 8.589 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

Statistics for this pull cycle (@devo_pulling_id=1674476113707):Number of requests made: 1; Number of events received: 12; Number of duplicated events filtered out: 0; Number of events generated and sent: 12; Average of events per second: 8.646.
2023-01-23T17:45:15.118    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,notifications,predefined) -> The data is up to date!

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Expand

title	Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_date_utc parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

Suspicious Activity service

Expand

title	Internal process and deduplication method

All of the notifications will be fetched from the API call, then based on the historic date provided or default historic days we will return notifications with a timestamp higher than the most recent timestamp for that Notification, we will update the state with the id’s and its log time which are polled after removing duplicates.
Hence in the next pull the ids present in the state file will not be pulled again and duplicates will be removed.

Expand

title	Devo categorization and destination

All notifications of Notifications service are ingested into the table myedr.appmalwarebytes.nebula.notifications.notification

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_eu_1) -> Starting thread
2023-01-23T17:45:11.730    INFO InputProcess::MainThread -> [GC] global: 28.3% -> 28.4%, process: RSS(46.75MiB -> 46.99MiB), VMS(791.24MiB -> 791.24MiB)
2023-01-23T17:45:11.751    INFO OutputProcess::MainThread -> [GC] global: 28.4% -> 28.4%, process: RSS(49.03MiB -> 49.44MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-23T17:45:12.477    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "139635964562208"
2023-01-23T17:45:12.478    INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "139635964560816"
2023-01-23T17:45:13.273    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Requesting access token from the Nebula server
2023-01-23T17:45:13.276    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 18:15:11
2023-01-23T17:45:13.277    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 
2023-01-23T17:45:13.278    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,notifications#predefined) -> Setup for module <NebulaNotificationsDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Starting data collection every 600 seconds
2023-01-23T22:23:03.544    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Pull Started
2023-01-23T22:23:05.259    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/sa
2023-01-23T22:23:05.264    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Removing the duplicate events if present...
2023-01-23T22:23:05.264    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Number of suspicious_activity sent to Devo: 0
2023-01-23T22:23:05.264    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Total number of suspicious_activity: 1
2023-01-23T22:23:05.265    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-23T22:23:05.265    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Saved state: {'last_polled_timestamp': 1674492783.536124, 'historic_date_utc': None, 'ids_with_same_timestamp': {}, '@persistence_version': 1}
2023-01-23T22:23:05.265    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674492783536):Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-01-23T22:23:05.265    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> The data is up to date!
2023-01-23T22:23:05.265    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> Data collection completed. Elapsed time: 1.730 seconds. Waiting for 598.270 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

Statistics for this pull cycle (@devo_pulling_id=1674492783536):Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-01-23T22:23:05.265    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> The data is up to date!

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Expand

title	Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_date_utc parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

Detection service

Expand

title	Internal process and deduplication method

All of the detections will be fetched from the API call. We will check if the ID is included in the state file after we get the list of detections. Then based on the historic date provided or default historic days we will return detection with a timestamp higher than the most recent timestamp for that detection, we will update the state with the IDs and their log time which are polled after removing duplicates.
Hence in the next PUll the IDs present in the state file will not be pulled again and duplicates will be removed.

Expand

title	Devo categorization and destination

All detection of Detection service is ingested into the table myedr.appmalwarebytes.nebula.detectionsdetection.

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_eu_1) -> Starting thread
2023-01-24T08:08:45.177    INFO InputProcess::MainThread -> <rate_limiter> setting has been accepted with the content {'period_in_seconds': 60, 'requests_limit_in_units': 25}
2023-01-24T08:08:45.177 WARNING InputProcess::MainThread -> The rate_limiter object has been overridden with the following config: {'period_in_seconds': 60, 'requests_limit_in_units': 25}
2023-01-24T08:08:45.178    INFO InputProcess::MainThread -> Running custom validation rules
2023-01-24T08:08:45.178    INFO InputProcess::MainThread -> Running custom init variables validation
2023-01-24T08:08:45.183    INFO InputProcess::MainThread -> Custom init variables validation completed
2023-01-24T08:08:45.183    INFO InputProcess::MainThread -> NebulaDetectionDataPuller(example_input,12345,detections,predefined) Finalizing the execution of init_variables()
2023-01-24T08:08:45.185    INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s)
2023-01-24T08:08:45.186    INFO InputProcess::MainThread -> ServiceThread(example_input,12345,detections,predefined) - Starting thread (execution_period=60s)
2023-01-24T08:08:45.186    INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Starting thread
2023-01-24T08:08:45.187    INFO InputProcess::MainThread -> NebulaDetectionDataPuller(example_input,12345,detections,predefined) - Starting thread
2023-01-24T08:08:45.187 WARNING InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Waiting until setup will be executed
2023-01-24T08:08:45.188    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Token has expired. Generating the new one
2023-01-24T08:08:45.188 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> The token/header/authentication is expired and it needs to be refreshed
2023-01-24T08:08:45.189    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Requesting access token from the Nebula server
2023-01-24T08:08:45.200    INFO OutputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(46.68MiB -> 48.02MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-24T08:08:45.218    INFO InputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(47.50MiB -> 47.52MiB), VMS(791.47MiB -> 791.47MiB)
2023-01-24T08:08:45.524    INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140547371149536"
2023-01-24T08:08:45.532    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140547385450448"
2023-01-24T08:08:46.347    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Requesting access token from the Nebula server
2023-01-24T08:08:46.349    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-24 08:38:45
2023-01-24T08:08:46.350    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 
2023-01-24T08:08:46.350    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined) -> Setup for module <NebulaDetectionDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Pull Started
2023-01-24T08:08:50.880    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/export
2023-01-24T08:08:50.883    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Received 650 detection ids from Nebula , fetching information for each detection id....
2023-01-24T08:08:50.936    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:08:50.937    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fa2d7c5f-ffd3-4ad6-8ab0-b703d30c0807 wait for a while ...
2023-01-24T08:08:52.605    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fa2d7c5f-ffd3-4ad6-8ab0-b703d30c0807
2023-01-24T08:08:52.606    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:08:52.606    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 62039c92-f50a-48c3-bd96-94364d76ec2b wait for a while ...
2023-01-24T08:08:54.247    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/62039c92-f50a-48c3-bd96-94364d76ec2b
2023-01-24T08:08:54.247    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:08:54.248    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 15828462-55a0-4f4c-80ea-596b9b66a2bb wait for a while ...
2023-01-24T08:08:55.719    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/15828462-55a0-4f4c-80ea-596b9b66a2bb
2023-01-24T08:08:55.720    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:08:55.721    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 0397d58f-5815-494d-9ecf-76cc8fdb259d wait for a while ...
2023-01-24T08:08:57.213    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/0397d58f-5815-494d-9ecf-76cc8fdb259d
2023-01-24T08:08:57.214    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:08:57.214    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 6bd1f1fd-7c40-401c-8fc5-e66957dd506f wait for a while ...
2023-01-24T08:08:58.750    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/6bd1f1fd-7c40-401c-8fc5-e66957dd506f
2023-01-24T08:08:58.751    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:08:58.752    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = c61ee947-e1c0-4fb2-b32a-a48c107b71fe wait for a while ...
2023-01-24T08:09:00.286    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/c61ee947-e1c0-4fb2-b32a-a48c107b71fe
2023-01-24T08:09:00.286    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:00.287    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = ee255a83-1c30-49b5-966e-e8baeb7e7c1d wait for a while ...
2023-01-24T08:09:01.721    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/ee255a83-1c30-49b5-966e-e8baeb7e7c1d
2023-01-24T08:09:01.721    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:01.722    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 3b1ff58d-88b6-47df-a2f0-4d4416b0a476 wait for a while ...
2023-01-24T08:09:03.153    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/3b1ff58d-88b6-47df-a2f0-4d4416b0a476
2023-01-24T08:09:03.153    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:03.154    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 3d9295a2-25a9-4f75-89e1-b146249efe71 wait for a while ...
2023-01-24T08:09:04.996    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/3d9295a2-25a9-4f75-89e1-b146249efe71
2023-01-24T08:09:04.996    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:04.997    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 40c6dca2-094a-43df-b339-36b71bd4d548 wait for a while ...
2023-01-24T08:09:06.533    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/40c6dca2-094a-43df-b339-36b71bd4d548
2023-01-24T08:09:06.533    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:06.534    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 20804a4a-6d3a-4c69-b7b6-dc64638f040f wait for a while ...
2023-01-24T08:09:08.171    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/20804a4a-6d3a-4c69-b7b6-dc64638f040f
2023-01-24T08:09:08.171    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:08.172    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = bb59bdb2-9eb3-4250-9380-b97186491063 wait for a while ...
2023-01-24T08:09:09.399    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/bb59bdb2-9eb3-4250-9380-b97186491063
2023-01-24T08:09:09.399    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:09.400    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = ca3b45af-2632-4f1f-a40c-4fdd02dead60 wait for a while ...
2023-01-24T08:09:10.835    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/ca3b45af-2632-4f1f-a40c-4fdd02dead60
2023-01-24T08:09:10.836    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:10.837    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 33139f41-395a-48d4-9949-dec3813cc62b wait for a while ...
2023-01-24T08:09:12.164    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/33139f41-395a-48d4-9949-dec3813cc62b
2023-01-24T08:09:12.164    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:12.165    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8ab05aa3-e732-4506-bf67-bd67bf1e9c8a wait for a while ...
2023-01-24T08:09:13.594    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8ab05aa3-e732-4506-bf67-bd67bf1e9c8a
2023-01-24T08:09:13.594    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:13.594    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 737d64e4-9f7c-4da0-95cc-db7478802979 wait for a while ...
2023-01-24T08:09:14.970    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/737d64e4-9f7c-4da0-95cc-db7478802979
2023-01-24T08:09:14.970    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:14.971    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 93967a0c-edee-425a-ab83-a5f8be36e97e wait for a while ...
2023-01-24T08:09:16.670    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/93967a0c-edee-425a-ab83-a5f8be36e97e
2023-01-24T08:09:16.671    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:16.672    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 0216543f-2a7e-4898-8ac9-ede98c6e3d01 wait for a while ...
2023-01-24T08:09:18.308    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/0216543f-2a7e-4898-8ac9-ede98c6e3d01
2023-01-24T08:09:18.309    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:18.309    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 022ce235-4eda-43a4-bd92-ba6934376554 wait for a while ...
2023-01-24T08:09:19.846    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/022ce235-4eda-43a4-bd92-ba6934376554
2023-01-24T08:09:19.848    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:19.849    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 91834b2d-1c97-469e-8cd7-81385683a41e wait for a while ...
2023-01-24T08:09:21.377    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/91834b2d-1c97-469e-8cd7-81385683a41e
2023-01-24T08:09:21.377    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:21.378    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 53c10041-07da-401b-bfbb-a39671b88d18 wait for a while ...
2023-01-24T08:09:22.814    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/53c10041-07da-401b-bfbb-a39671b88d18
2023-01-24T08:09:22.814    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:22.815    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fb6c8138-9cda-49a3-89fb-dcf20b1133f2 wait for a while ...
2023-01-24T08:09:24.145    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fb6c8138-9cda-49a3-89fb-dcf20b1133f2
2023-01-24T08:09:24.145    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:24.146    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 236fd0f5-69a9-4663-88dd-a7559604bd79 wait for a while ...
2023-01-24T08:09:25.575    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/236fd0f5-69a9-4663-88dd-a7559604bd79
2023-01-24T08:09:25.576    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:25.576    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 51567b63-de25-4a9e-b62e-c404b4fc3119 wait for a while ...
2023-01-24T08:09:27.012    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/51567b63-de25-4a9e-b62e-c404b4fc3119
2023-01-24T08:09:27.013    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:27.013    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 4d4e9dbc-f6a2-44e7-a246-a5f98f36f706 wait for a while ...
2023-01-24T08:09:28.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/4d4e9dbc-f6a2-44e7-a246-a5f98f36f706
2023-01-24T08:09:28.550    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:28.550    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 11cdbe65-d5b9-4051-b2f1-42c7c7fa671f wait for a while ...
2023-01-24T08:09:29.983    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/11cdbe65-d5b9-4051-b2f1-42c7c7fa671f
2023-01-24T08:09:29.984    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:29.985    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 825e5fca-8b97-4108-a4c7-621bc122101b wait for a while ...
2023-01-24T08:09:31.416    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/825e5fca-8b97-4108-a4c7-621bc122101b
2023-01-24T08:09:31.417    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:31.418    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = df1cde2f-6b01-4912-ab66-d73e44d048a5 wait for a while ...
2023-01-24T08:09:32.849    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/df1cde2f-6b01-4912-ab66-d73e44d048a5
2023-01-24T08:09:32.851    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:32.852    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 97074472-38ba-4a2c-933e-56fa4716d369 wait for a while ...
2023-01-24T08:09:34.215    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/97074472-38ba-4a2c-933e-56fa4716d369
2023-01-24T08:09:34.217    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:34.218    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 7c8a7370-1051-4180-9552-5c986d158ff1 wait for a while ...
2023-01-24T08:09:35.615    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/7c8a7370-1051-4180-9552-5c986d158ff1
2023-01-24T08:09:35.616    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:35.616    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f6b390f7-f4fe-4ab9-946e-b4283b8e508b wait for a while ...
2023-01-24T08:09:37.049    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f6b390f7-f4fe-4ab9-946e-b4283b8e508b
2023-01-24T08:09:37.051    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:37.052    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = b0a299c3-a541-487b-ac21-0344f99eb2de wait for a while ...
2023-01-24T08:09:38.385    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/b0a299c3-a541-487b-ac21-0344f99eb2de
2023-01-24T08:09:38.386    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:38.386    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 22020f80-53ec-4095-9c8b-58cdefe53903 wait for a while ...
2023-01-24T08:09:39.812    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/22020f80-53ec-4095-9c8b-58cdefe53903
2023-01-24T08:09:39.813    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:39.813    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 34c2bb2c-1445-46b1-8e62-890f52723478 wait for a while ...
2023-01-24T08:09:41.140    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/34c2bb2c-1445-46b1-8e62-890f52723478
2023-01-24T08:09:41.140    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:41.140    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f358ea21-fd91-4577-a1a7-32dbdae6c2f9 wait for a while ...
2023-01-24T08:09:42.503    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f358ea21-fd91-4577-a1a7-32dbdae6c2f9
2023-01-24T08:09:42.504    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:42.504    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = ba4d297e-6605-4b4e-93e0-dd01cdf78e3f wait for a while ...
2023-01-24T08:09:43.909    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/ba4d297e-6605-4b4e-93e0-dd01cdf78e3f
2023-01-24T08:09:43.911    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:43.912    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 4cb1e887-6893-4aad-92bc-236ad272b143 wait for a while ...
2023-01-24T08:09:45.218    INFO OutputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(49.73MiB -> 49.73MiB), VMS(1.20GiB -> 1.20GiB)
2023-01-24T08:09:45.233    INFO InputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(51.98MiB -> 51.98MiB), VMS(792.35MiB -> 792.35MiB)
2023-01-24T08:09:45.234    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> Consumed messages: 41, total_bytes: 28235 (60.059315 seconds)
2023-01-24T08:09:45.234    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 41 messages 
(60.061175 seconds) => 0 msg/sec
2023-01-24T08:09:45.445    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/4cb1e887-6893-4aad-92bc-236ad272b143
2023-01-24T08:09:45.446    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:45.446    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 95e4edc2-49e5-4ed5-b65b-1b0c56d039b0 wait for a while ...
2023-01-24T08:09:47.144    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/95e4edc2-49e5-4ed5-b65b-1b0c56d039b0
2023-01-24T08:09:47.145    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:47.145    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 46bc8fa8-801d-4f1a-b43e-6ab90fec9843 wait for a while ...
2023-01-24T08:09:48.520    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/46bc8fa8-801d-4f1a-b43e-6ab90fec9843
2023-01-24T08:09:48.522    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:48.524    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 3b88d4f9-edf6-4aee-bee1-26ef90fd112b wait for a while ...
2023-01-24T08:09:50.053    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/3b88d4f9-edf6-4aee-bee1-26ef90fd112b
2023-01-24T08:09:50.055    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:50.056    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 84f490e9-de8e-4b02-b243-5fd7b84e2d6f wait for a while ...
2023-01-24T08:09:51.692    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/84f490e9-de8e-4b02-b243-5fd7b84e2d6f
2023-01-24T08:09:51.693    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:51.694    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 7cec1216-13b2-49c7-8871-c2342f9f021a wait for a while ...
2023-01-24T08:09:53.130    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/7cec1216-13b2-49c7-8871-c2342f9f021a
2023-01-24T08:09:53.132    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:53.133    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 14ec5bbe-52ca-43c3-a946-4027890863a7 wait for a while ...
2023-01-24T08:09:54.560    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/14ec5bbe-52ca-43c3-a946-4027890863a7
2023-01-24T08:09:54.560    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:54.561    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = c16c6dd8-3afc-4d57-8867-41000062f673 wait for a while ...
2023-01-24T08:09:55.995    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/c16c6dd8-3afc-4d57-8867-41000062f673
2023-01-24T08:09:55.996    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:55.997    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching
information for particular id = 6e67790c-79fd-41f2-8b45-17a88995d1a2 wait for a while ...
2023-01-24T08:09:57.630    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/6e67790c-79fd-41f2-8b45-17a88995d1a2
2023-01-24T08:09:57.631    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:57.631    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8858d65a-a593-4496-8fc2-a80f612470ac wait for a while ...
2023-01-24T08:09:59.070    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8858d65a-a593-4496-8fc2-a80f612470ac
2023-01-24T08:09:59.071    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:59.071    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = aed74e91-3c84-4990-bf50-d724da00b134 wait for a while ...
2023-01-24T08:10:00.503    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/aed74e91-3c84-4990-bf50-d724da00b134
2023-01-24T08:10:00.504    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:00.504    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 814f10c8-ccd0-4f69-923f-eb5a7111ea07 wait for a while ...
2023-01-24T08:10:01.936    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/814f10c8-ccd0-4f69-923f-eb5a7111ea07
2023-01-24T08:10:01.936    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:01.936    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = d32f5f0d-0046-4045-9016-04d16313a112 wait for a while ...
2023-01-24T08:10:03.473    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/d32f5f0d-0046-4045-9016-04d16313a112
2023-01-24T08:10:03.473    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:03.473    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fe5479f7-ef04-4b1b-915e-76dc98770ee8 wait for a while ...
2023-01-24T08:10:04.905    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fe5479f7-ef04-4b1b-915e-76dc98770ee8
2023-01-24T08:10:04.907    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:04.909    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 364d231e-35cb-48ff-a3d4-28b3da6d97a5 wait for a while ...
2023-01-24T08:10:06.287    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/364d231e-35cb-48ff-a3d4-28b3da6d97a5
2023-01-24T08:10:06.288    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:06.288    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3 wait for a while ...
2023-01-24T08:10:07.778    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3
2023-01-24T08:10:07.778    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:07.779    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching
information for particular id = 6e67790c-79fd-41f2-8b45-17a88995d1a2 wait for a while ...
2023-01-24T08:09:57.630    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/6e67790c-79fd-41f2-8b45-17a88995d1a2
2023-01-24T08:09:57.631    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:57.631    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8858d65a-a593-4496-8fc2-a80f612470ac wait for a while ...
2023-01-24T08:09:59.070    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8858d65a-a593-4496-8fc2-a80f612470ac
2023-01-24T08:09:59.071    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:09:59.071    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = aed74e91-3c84-4990-bf50-d724da00b134 wait for a while ...
2023-01-24T08:10:00.503    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/aed74e91-3c84-4990-bf50-d724da00b134
2023-01-24T08:10:00.504    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:00.504    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 814f10c8-ccd0-4f69-923f-eb5a7111ea07 wait for a while ...
2023-01-24T08:10:01.936    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/814f10c8-ccd0-4f69-923f-eb5a7111ea07
2023-01-24T08:10:01.936    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:01.936    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = d32f5f0d-0046-4045-9016-04d16313a112 wait for a while ...
2023-01-24T08:10:03.473    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/d32f5f0d-0046-4045-9016-04d16313a112
2023-01-24T08:10:03.473    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:03.473    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = fe5479f7-ef04-4b1b-915e-76dc98770ee8 wait for a while ...
2023-01-24T08:10:04.905    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/fe5479f7-ef04-4b1b-915e-76dc98770ee8
2023-01-24T08:10:04.907    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:04.909    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 364d231e-35cb-48ff-a3d4-28b3da6d97a5 wait for a while ...
2023-01-24T08:10:06.287    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/364d231e-35cb-48ff-a3d4-28b3da6d97a5
2023-01-24T08:10:06.288    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:06.288    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3 wait for a while ...
2023-01-24T08:10:07.778    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3
2023-01-24T08:10:07.778    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:07.779    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching 
information for particular id = 99f714b6-85f3-41ce-96fa-f1f8aa675a88 wait for a while ...
2023-01-24T08:10:09.308    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/99f714b6-85f3-41ce-96fa-f1f8aa675a88
2023-01-24T08:10:09.309    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:09.309    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = aa7c17cd-940f-4b0f-a494-9e7f2a738a8c wait for a while ...
2023-01-24T08:10:10.744    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/aa7c17cd-940f-4b0f-a494-9e7f2a738a8c
2023-01-24T08:10:10.745    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:10.745    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = d3084825-954b-4b1d-aec9-c02b95a412e8 wait for a while ...
2023-01-24T08:10:12.176    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/d3084825-954b-4b1d-aec9-c02b95a412e8
2023-01-24T08:10:12.176    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:12.176    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Fetching information for particular id = 8365f56e-eb28-40a1-a14c-4380587f87ef wait for a while ...
2023-01-24T08:10:13.709    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections/8365f56e-eb28-40a1-a14c-4380587f87ef
2023-01-24T08:10:13.710    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detection if present...
2023-01-24T08:10:13.719    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Number of detections sent to Devo: 56
2023-01-24T08:10:13.721    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-24T08:10:13.722    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Saved state: {'last_polled_timestamp': 1674527927.190399, 'historic_date_utc': 1674461577.422, 'detection_ids': {'fa2d7c5f-ffd3-4ad6-8ab0-b703d30c0807': '2023-01-24T01:55:12.35726Z', '62039c92-f50a-48c3-bd96-94364d76ec2b': '2023-01-24T01:54:00.435667Z', '15828462-55a0-4f4c-80ea-596b9b66a2bb': '2023-01-24T01:52:48.431457Z', '0397d58f-5815-494d-9ecf-76cc8fdb259d': '2023-01-24T01:32:24.58344Z', '6bd1f1fd-7c40-401c-8fc5-e66957dd506f': '2023-01-24T01:00:29.734536Z', 'c61ee947-e1c0-4fb2-b32a-a48c107b71fe': '2023-01-24T00:49:23.387871Z', 'ee255a83-1c30-49b5-966e-e8baeb7e7c1d': '2023-01-24T00:31:59.350819Z', '3b1ff58d-88b6-47df-a2f0-4d4416b0a476': '2023-01-24T00:30:47.430603Z', '3d9295a2-25a9-4f75-89e1-b146249efe71': '2023-01-24T00:29:35.465086Z', '40c6dca2-094a-43df-b339-36b71bd4d548': '2023-01-24T00:11:01.123421Z', '20804a4a-6d3a-4c69-b7b6-dc64638f040f': '2023-01-23T23:57:41.332863Z', 'bb59bdb2-9eb3-4250-9380-b97186491063': '2023-01-23T23:37:25.151688Z', 'ca3b45af-2632-4f1f-a40c-4fdd02dead60': '2023-01-23T23:14:49.155606Z', '33139f41-395a-48d4-9949-dec3813cc62b': '2023-01-23T23:14:05.365414Z', '8ab05aa3-e732-4506-bf67-bd67bf1e9c8a': '2023-01-23T23:08:50.457342Z', '737d64e4-9f7c-4da0-95cc-db7478802979': '2023-01-23T23:07:39.026314Z', '93967a0c-edee-425a-ab83-a5f8be36e97e': '2023-01-23T23:06:29.931750Z', '0216543f-2a7e-4898-8ac9-ede98c6e3d01': '2023-01-23T22:15:56.204348Z', '022ce235-4eda-43a4-bd92-ba6934376554': '2023-01-23T22:04:35.363259Z', '91834b2d-1c97-469e-8cd7-81385683a41e': '2023-01-23T22:04:19.1193Z', '53c10041-07da-401b-bfbb-a39671b88d18': '2023-01-23T21:40:59.057789Z', 'fb6c8138-9cda-49a3-89fb-dcf20b1133f2': '2023-01-23T21:13:08.316732Z', '236fd0f5-69a9-4663-88dd-a7559604bd79': '2023-01-23T20:50:48.680785Z', '51567b63-de25-4a9e-b62e-c404b4fc3119': '2023-01-23T20:40:56.062104Z', '4d4e9dbc-f6a2-44e7-a246-a5f98f36f706': '2023-01-23T20:35:00.208627Z', '11cdbe65-d5b9-4051-b2f1-42c7c7fa671f': '2023-01-23T20:07:41.059141Z', '825e5fca-8b97-4108-a4c7-621bc122101b': '2023-01-23T19:55:57.170841Z', 'df1cde2f-6b01-4912-ab66-d73e44d048a5': '2023-01-23T19:39:23.335462Z', '97074472-38ba-4a2c-933e-56fa4716d369': '2023-01-23T19:14:50.820054Z', '7c8a7370-1051-4180-9552-5c986d158ff1': '2023-01-23T19:10:20.571464Z', 'f6b390f7-f4fe-4ab9-946e-b4283b8e508b': '2023-01-23T19:02:22.163271Z', 'b0a299c3-a541-487b-ac21-0344f99eb2de': '2023-01-23T18:53:12.388744Z', '22020f80-53ec-4095-9c8b-58cdefe53903': '2023-01-23T18:48:15.121402Z', '34c2bb2c-1445-46b1-8e62-890f52723478': '2023-01-23T17:56:25.103127Z', 'f358ea21-fd91-4577-a1a7-32dbdae6c2f9': '2023-01-23T17:36:35.874122Z', 'ba4d297e-6605-4b4e-93e0-dd01cdf78e3f': '2023-01-23T16:35:09.764540Z', '4cb1e887-6893-4aad-92bc-236ad272b143': '2023-01-23T16:24:38.448483Z', '95e4edc2-49e5-4ed5-b65b-1b0c56d039b0': '2023-01-23T16:04:07.716107Z', '46bc8fa8-801d-4f1a-b43e-6ab90fec9843': '2023-01-23T15:59:29.944425Z', '3b88d4f9-edf6-4aee-bee1-26ef90fd112b': '2023-01-23T15:38:52.531519Z', '84f490e9-de8e-4b02-b243-5fd7b84e2d6f': '2023-01-23T15:34:38.225593Z', '7cec1216-13b2-49c7-8871-c2342f9f021a': '2023-01-23T15:24:50.043742Z', '14ec5bbe-52ca-43c3-a946-4027890863a7': '2023-01-23T15:01:11.03246Z', 'c16c6dd8-3afc-4d57-8867-41000062f673': '2023-01-23T13:58:11.770177Z', '6e67790c-79fd-41f2-8b45-17a88995d1a2': '2023-01-23T13:36:57.367930Z', '8858d65a-a593-4496-8fc2-a80f612470ac': '2023-01-23T13:12:11.576420Z', 'aed74e91-3c84-4990-bf50-d724da00b134': '2023-01-23T13:00:32.356141Z', '814f10c8-ccd0-4f69-923f-eb5a7111ea07': '2023-01-23T12:50:53.842600Z', 'd32f5f0d-0046-4045-9016-04d16313a112': '2023-01-23T11:48:35.971445Z', 'fe5479f7-ef04-4b1b-915e-76dc98770ee8': '2023-01-23T11:42:34.132893Z', '364d231e-35cb-48ff-a3d4-28b3da6d97a5': '2023-01-23T11:38:02.863276Z', 'f1c43cd4-b39c-4e8c-80f3-e7da215ea3e3': '2023-01-23T11:22:28.967188Z', '99f714b6-85f3-41ce-96fa-f1f8aa675a88': '2023-01-23T11:17:27.70895Z', 'aa7c17cd-940f-4b0f-a494-9e7f2a738a8c': '2023-01-23T09:31:37.432437Z', 'd3084825-954b-4b1d-aec9-c02b95a412e8': '2023-01-23T09:05:18.292574Z', '8365f56e-eb28-40a1-a14c-4380587f87ef': '2023-01-23T08:36:38.65638Z'}, '@persistence_version': 1}
2023-01-24T08:10:13.722    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527927190):Number of requests made: 1; Number of events received: 650; Number of duplicated events filtered out: 594; Number of events generated and sent: 56; Average of events per second: 0.647.
2023-01-24T08:10:13.723    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> The data is up to date!
2023-01-24T08:10:13.724    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Data collection completed. Elapsed time: 86.534 seconds. Waiting for 513.466 second(s) until the next one
2023-01-24T08:10:13.729    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> Consumed messages: 3, total_bytes: 1896 (88.558356 seconds)

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

2023-01-24T08:10:13.722    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527927190):Number of requests made: 1; Number of events received: 650; Number of duplicated events filtered out: 594; Number of events generated and sent: 56; Average of events per second: 0.647.
2023-01-24T08:10:13.723    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> The data is up to date!

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Expand

title	Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_date_utc parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

DNS logs service

Expand

title	Internal process and deduplication method

All of the detections will be fetched from the API call. We will check if the ID is included in the state file after we get the list of detections. Then based on the historic date provided or default historic days we will return detection with a timestamp higher than the most recent timestamp for that detection, we will update the state with the IDs and their log time which are polled after removing duplicates.
Hence in the next PUll the IDs present in the state file will not be pulled again and duplicates will be removed.

Expand

title	Devo categorization and destination

All detection of Detection service is ingested into the table myedr.appmalwarebytes.nebula.detectionsdetection.

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

I2023-01-24T08:07:45.735    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-01-24T08:07:45.736    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Starting thread (every 300 seconds)
2023-01-24T08:07:45.736    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_eu_1) -> Starting thread
2023-01-24T08:07:45.738    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-01-24T08:07:45.738    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Starting thread (every 300 seconds)
2023-01-24T08:07:45.739    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_eu_1) -> Starting thread
2023-01-24T08:07:45.740    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-01-24T08:07:45.740    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Starting thread (every 300 seconds)
2023-01-24T08:07:45.741    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_eu_1) -> Starting thread
2023-01-24T08:07:45.741    INFO InputProcess::MainThread -> <rate_limiter> setting has been accepted with the content {'period_in_seconds': 60, 'requests_limit_in_units': 25}
2023-01-24T08:07:45.741 WARNING InputProcess::MainThread -> The rate_limiter object has been overridden with the following config: {'period_in_seconds': 60, 'requests_limit_in_units': 25}
2023-01-24T08:07:45.742    INFO InputProcess::MainThread -> Running custom validation rules
2023-01-24T08:07:45.742    INFO InputProcess::MainThread -> Running custom init variables validation
2023-01-24T08:07:45.747    INFO InputProcess::MainThread -> Custom init variables validation completed
2023-01-24T08:07:45.747    INFO InputProcess::MainThread -> NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) Finalizing the execution of init_variables()
2023-01-24T08:07:45.751    INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s)
2023-01-24T08:07:45.751    INFO InputProcess::MainThread -> ServiceThread(example_input,12345,dns_log_data,predefined) - Starting thread (execution_period=60s)
2023-01-24T08:07:45.752    INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Starting thread
2023-01-24T08:07:45.753    INFO InputProcess::MainThread -> NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) - Starting thread
2023-01-24T08:07:45.753    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Token has expired. Generating the new one
2023-01-24T08:07:45.754 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> The token/header/authentication is expired and it needs to be refreshed
2023-01-24T08:07:45.754 WARNING InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Waiting until setup will be executed
2023-01-24T08:07:45.755    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Requesting access token from the Nebula server
2023-01-24T08:07:45.772    INFO OutputProcess::MainThread -> [GC] global: 18.3% -> 18.4%, process: RSS(47.02MiB -> 47.86MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-24T08:07:45.787    INFO InputProcess::MainThread -> [GC] global: 18.4% -> 18.4%, process: RSS(47.08MiB -> 47.17MiB), VMS(791.65MiB -> 791.60MiB)
2023-01-24T08:07:46.087    INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140013684891072"
2023-01-24T08:07:46.088    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140013684750560"
2023-01-24T08:07:47.784    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Requesting access token from the Nebula server
2023-01-24T08:07:47.786    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-24 08:37:45
2023-01-24T08:07:47.786    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 
2023-01-24T08:07:47.786    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,dns_log_data#predefined) -> Setup for module <NebulaDnsLogsDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

2023-01-24T08:07:58.464    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/dns
2023-01-24T08:07:58.537    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Total Dns available are = 18527
2023-01-24T08:07:58.537    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Received 2000 dns data from Nebula Server
2023-01-24T08:07:58.537    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Removing the duplicate dns if present...
2023-01-24T08:07:58.545    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-24T08:07:58.545    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Received 317 dns data from Nebula Server after removing duplicates
2023-01-24T08:07:58.560    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Number of DNS logs sent to Devo: 317
2023-01-24T08:07:58.561    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Saved state: {'last_polled_timestamp': 1674527868.758331, 'historic_date_utc': 1674461577.422, 'next_cursor': 'WzE2NzQxMTUwMDM3NzgsImU2MjI3Y2NkLTE0OTItNDQxOC05MThjLWQ1NTRjNDg1ZWQ5YSJd', 'dns_ids': ['1de0c0c0-e88b-461f-806a-e9514a4a190e', '27ee777d-bf62-4f9b-a437-7359427d1283', 'eb3a915e-e957-4142-9b0a-fc1a6de6cdb3', 'f4e52927-0431-4da6-b05b-4dd08d99912b', 'd194dd03-c501-40c9-87c5-146298491e08', '25fa78a0-c932-4aa6-8f5b-027eca3adc19', '8c85017a-67d9-42ca-8ab0-75b585d349f6', 'eeb1e594-e9dc-42ab-9f4f-9505bd0f1af8', 'b10301d1-baf9-4894-a09d-4073ee16115d', '9873f89d-987f-46d9-980e-338607822957', '7ea98f40-2ec4-4718-a331-983476bcfdc7', '2a479e96-4faa-4ead-a1c9-3ee049c8c4a7', '28cdba37-c277-4203-aaab-4a6b5f975f38', '2b5f36d5-7d09-4c2b-aeb4-c216545f9dc5', '303fa343-dd53-4707-af6d-75054ad8c119', '47a5edae-576c-4d7d-b587-751e719bca60', '566e42c1-297b-470e-9b0b-ade0b49e2d92', 'ba9b9196-1a9d-4b38-9090-3eb8c0c61e02', 'be8d149c-a7b6-4a58-8573-96942f3f6d63', 'd559d57d-8c71-4089-9cca-4deae46f63d3', '5515c005-f916-4dd7-8ca5-42bd80f4cc82', '8d478713-ad1b-4f52-8bbf-1338120ea93d', 'd7f10f1f-7b26-4c11-a118-56b29235e197', '18d509ae-4474-450f-be00-729a11d21f58', '3dcff515-4a03-4b8d-80c0-9c089e1fc071', 'b9d9faed-4e1e-46b5-b8f0-7cf29076760d', 'bae5c67a-7544-47fc-82ab-1f64a15acc62', '2777c4f8-b38b-4055-bed8-6f8a6121687a', 'e3d25322-95fe-4f72-9ee6-8ed6097d1b20', '094cdf61-c755-4e0d-bf23-bccfe8a9b976', 'dfd154de-b5c5-463c-9191-3632fd62a757', '41c08650-c4c7-4b9b-acd0-f7db8eaa717e', '32781ab6-af42-4e05-b3f0-da75c07bee16', '3e8cb307-97e0-44b3-9c85-97e9793b6cc4', '79cff0e3-4fe4-4a81-a58f-562fc43f2230', '84404282-ba53-47ce-bbda-8d62a02d8ddc', 'af7dc4b1-0f49-4e44-ad7b-14dee88302ee', 'b96613b8-47fc-4c55-a31b-83eb4b5b53d3', 'd25fcaf3-a466-4361-986b-aa6e4d626b24', 'd4ad6765-45bf-4ba5-9b57-e69eafb84293', '028f0591-3dfc-4a23-8a34-7dbabf7878ac', '9090c00a-cf96-496c-a923-9a97a4db68c0', 'a20439b7-3f57-4f32-b733-573e217401b1', '3d038e99-3de9-40db-adfe-c8110effdf15', '1eafeacb-1c22-4841-8349-8db5f89c5c91', 'ffc8f64a-ebbb-436b-a35e-b4d2551bc76a', 'e27e3def-371b-496c-8da4-6da680b3e9aa', '3d53887b-7c20-4d95-a721-e990b27204f0', '45614a51-9150-4b29-add2-c931e87b687d', '48a1dfc7-810e-46c2-b82c-0497471557fe', '26103bff-524f-4b5b-a063-c5e0a6ab55f5', '78b94735-b9ff-479f-bbfb-62b1ebfcdd1e', '0bf14e71-7e47-438b-9159-5b89d0705249', 'd7d78b1f-321f-4610-b013-14e77a54e69a', 'ef48a404-caf6-4abf-9737-0bb82f5a9a31', '00e9dfbb-d816-433b-a58b-5aed9358a3ea', '089d1139-3fcc-4d7a-83f5-469dd221d7f1', '3228d1ba-639f-4cef-8af5-8282efe9f504', 'a57a3d7d-94b1-4c2c-86e7-e63f92aa47da', '68602fb1-f79d-4369-a52a-247edb4169b6', 'ff7cfd49-6e1e-4ef6-bcbf-bd44be169295', '1282c2dc-f149-411b-8979-65cf2037550d', '5c9a046b-2a49-4488-b95d-992470bdc3a9', '6246e76e-c89b-4660-a1bc-b590908cc0ab', '7702c06d-6181-48d9-8aff-03d1bdcdeca1', 'c290cc60-48c5-4b96-b541-84a259ff3a65', 'c98f7946-5b0f-4f49-854a-a2180d36b717', 'e51f976c-1ab5-4f5f-98f4-c4d60c60e224', 'f6735a22-9d36-420a-a287-579cf6325266', 'fb8d128f-1c98-44fa-a947-8ec768916740', 'dc120dbe-5eac-4c9b-bedd-a44c8c2536a2', '46f456b1-1d6a-4b3a-874c-6b7fa4c19d67', '6638158c-1f83-482b-9c34-0c9b8aae911b', 'a9f2f3b0-78f7-48a5-b4ac-e1bcb671b9de', 'fbf617a1-0e7c-4241-986f-445f7a141471', '567822c9-6589-4cd3-9b6e-b25fab30098f', 'e9e2fe0d-dc86-414d-afe3-d6e53f8eb0bf', '0173cddb-506c-495f-9aac-5731d8166688', '6ec46c16-602c-496c-a203-53e27474b455', '758805cc-37c3-4d80-8301-41e42e92f6ca', 'a8bf6187-0b3b-430a-a0c6-da802d28ad78', 'd0393382-97be-480e-949f-fd080087b09d', '1a00de7a-c59f-4531-b269-9796d92f52c7', '1f7cc29b-a87c-48d5-a4ab-48c43e619753', '44501eee-4930-4659-a978-2dfc29a9aa03', '7527e921-7c5b-4bd4-aa3e-b61b51e01888', 'af336c12-5e70-463a-84a5-1cd32b51d187', '9553af20-accc-48f3-a6ee-ded7932309d3', '9aeb2967-9e81-4f45-8239-f8c445eea2b5', '514996a1-1a12-4b67-b480-674b0d82699f', '86e2f394-5af8-4a18-a461-f63674d1bbba', '3aca057e-995b-42b0-b000-db4ac9e9a3d4', '48a7fe31-8fff-4f8f-87b7-b843e68370c9', '9cce8f79-2715-4c42-be1a-63753765d510', '6f5b6a20-9aab-45ea-9de1-57ce177aedd1', 'bb12c45b-b0d6-4ca1-a583-abdb92bd3dc8', 'c858beb0-9250-45a0-946d-4f21fc208787', '06d75995-f28b-491f-8cd1-b1fa9aba3428', 'dc45e1e0-9277-4b0d-b6f2-eab10b78d43b', '17f22108-1899-4aa6-8fd4-d41634312df0', '25bb57ca-1c25-4802-add8-9cdbb2996a41', '370fecb6-7be4-48c4-8b40-eb247c0be5ab', '778dc2db-57dd-4f92-8bed-fb74879b93ad', '8c4384d5-e322-4d0c-a65c-d93bd73a1187', 'b25a7ae1-1bf4-4b5a-8f8f-8e17c7993337', '046dc0a2-17fd-4e50-b1b0-ac55946869e2', 'a92c8d2b-4498-46d6-a571-1a4d843fe05d', 'e2910d04-f01e-4a86-82a8-c73c3e48052e', 'ef281b09-c857-4d5a-9c43-b1ff3ff37525', '8ffc027e-d943-4caa-8f95-39e91624c350', '4a628ddb-c15b-4ae8-9036-2782209871b2', '2b965c56-6270-4773-a9d2-e0b7460d5d6f', '17d8f338-9b68-4d73-8107-cf9e8dffbea0', 'e2ca53b8-f2c6-4688-847a-9f8c023cc32c', 'e6f47d19-44c2-4c71-80b4-d25e99b8cd37', '092b3af1-311b-42eb-b0f6-a0b5212f2fe2', '1f5662ca-2edf-49ff-841b-1e2ec1261a64', '88fc2227-c298-4384-824d-dca6b8392301', 'aeb14001-91c5-49b9-9f0c-9a434667a038', 'b214d443-782f-4253-b500-b744fb57d789', '2b04734e-f49d-44f1-83bb-8df1c14f3d29', '45fb4ac6-9cb4-4673-943d-a5dcca76bba7', '5c22aaf4-4d45-4b39-891e-e3c0fe463cfe', '96d439fe-c379-42c5-b0fc-483d96fb1a8f', 'b1f94e26-c8c1-4283-8546-52064c93da7a', '2eabc5b6-aa2b-40b9-b2d8-9b67b7a65313', '92df367a-9a7c-40d3-ae8a-97171027557b', 'b0c3947c-a207-46d8-aa1b-04dc71d4e800', 'ba00d2eb-579f-42b2-9ad1-3fc86676ca3f', '0338ce8e-d2c5-41e2-8222-88144ba2e856', 'e6481e76-59c9-448e-a904-47c63d0a0c05', '555404e3-71a2-4576-b05d-457205e7c4ee', 'f9833003-7d06-4f6b-8950-de29b7fb541c', '3c3987dd-1110-4772-9af0-c1ac86684e24', '3cf0aae3-fca2-4d71-be2f-169e7c6c6e57', 'd875284e-7c56-4f26-87f8-fc3c509d2f3a', '2f6dfeed-8e7e-4956-a76c-fe4e6403f84f', '76e3130c-8c60-45d1-93cb-de0c68a9456b', 'af2ecc60-92f9-48d6-8822-a53fd62951cc', '2ab8a63d-16ed-4f1b-9456-779b8429d549', '1185ab63-4125-42dd-a858-4970539b6bb5', '5f545e10-0fca-494f-8a28-faf26558b0c3', '6d3a71ff-6eff-46e9-8015-009c899adb2e', '7ab963fe-8d7b-4ff0-925b-deb22ab9ce09', '95b0bddf-5a77-47d2-88a3-e919a8dd096b', 'a2c15803-dc25-4e00-a40e-53453b3dd1a0', 'cf8dd265-dc2e-4f1d-af26-1f9a8e87455c', 'e68d15e5-d5ba-48d5-8358-20edb875a174', 'fec1838f-f01c-4dd2-afb5-0c9a48cb9729', '23c6c17e-15d8-4f76-96c7-c92649ee7578', 'c47c426f-a67c-4032-937f-b04b31ca339c', 'e320398c-5517-49b2-9a81-bb56dd38fe61', 'd9eb4741-dea0-48ad-81b4-6b08b1ae9e95', 'e77dec9f-642b-4bea-96c9-d1a87259e47a', '4a4f9cf6-47c6-4ad1-b008-4e59e8c0eb96', '1127f942-1212-4803-bcf5-f594efc2df04', '4e404e87-6df1-4a72-b38c-883c8da1471b', '64721995-4479-477f-9e45-261f4b222135', 'a439ac26-549b-43a1-84d7-8888ed21d5f7', '4596a4d3-5995-4a32-8f77-2e4b19a9bd1d', '7c5e3937-1ea5-4a56-8abd-e39836b2bb7c', 'ce3b6301-93a3-4d6c-9b3c-bc4536ac8b8d', '97865518-e9ae-42d4-8e78-23ac1e617d6f', 'f1669e02-83e9-49b4-bf0a-afa7b7984830', '639717bb-f340-48ac-8ea9-6b669213d67c', '2e66e679-73a4-4f51-8a5c-254f70699706', 'd8290d24-3b48-431f-85e8-e27734fb26fa', 'e5226a02-798c-40f4-a9a4-a5d233fb929f', 'da169995-2b4b-4c12-a44a-d7d6400f595a', 'bdb3bdf5-99a8-4e7c-976c-301783eaf0e6', '59ee190e-4065-4a8f-9dff-4ef7f9fadd92', 'f3372a3a-437a-4661-970a-36b66dbfa780', '84e93ff2-7ff9-4201-a389-71de887544c8', '91468473-98b2-4424-a8aa-b66ddf7aacf8', '9792149b-242a-4811-9454-edab76c35b3c', 'bc698860-4dd5-4298-b033-c35b4f6f2cb4', 'cd89a85d-760b-4c9f-b86c-29373cc6e96c', 'db0a0a99-25f6-4e28-a2b2-101109a89413', '37ff68e6-7b9b-4289-8ae2-44b2e7a5a6a6', 'd41df096-1c03-4f8d-a857-7b906e93aa56', '7524fa94-51ad-4563-8016-e409a8a66116', '20d1d647-88b1-4c1c-a663-cbe2d155ce5a', '35a01eaf-fa5b-44e3-bb24-bbff0bd045da', '3fc5f77e-826c-48c3-a377-b8b45d1a91f2', '480d10f9-e822-4129-a74d-def7ef909cce', '72f36c63-1115-4e5e-975e-d74541bfbeb3', 'ede029af-3387-47f2-8160-6e1c1b120864', 'ffe2e0c0-0990-49f9-b0a8-834a89e87d48', '0954de07-b776-4329-91e6-1cb46324136f', 'd5314366-c9cc-4c1a-b94a-70a7c7f46606', '28514cd5-08c2-4464-aa8a-d7b89ba1d12d', '97e2b0df-b611-4486-bbc3-62815dceb653', '13b2a0fe-41f9-4f2a-8dba-1cf0a8a250da', 'faa8fece-0d1a-4381-bb50-7cb2a04ddb99', '09a61b23-bd9e-4ee8-93ef-dd23f47517c8', '1c4a86cf-d159-4965-baf2-26659af45429', '8a982f4b-7dad-4554-a8a8-81ac7371bf97', '9c7b376f-beb9-48b9-940a-a72605fa20d8', '7c566512-6da7-4a13-8a05-7766cc49ee6a', '839030d0-54a2-4c96-8849-ff2ed4c91814', 'a4c19357-d612-4717-83b1-a7fe153b4196', 'ac134c4b-a660-4104-b7d0-f07b2960e432', 'bc7e5ef2-1844-4cfd-a0e0-d72094a7d5af', '94b06864-d5a0-4109-ada3-076167e094da', '2707c4f8-f323-4a31-bfcb-5750fd1e67c9', 'a5134a8d-d287-4dca-a628-e063e20c91f1', 'bfe454f3-fe1d-42ac-ae4e-8a5b671d80df', '045feb3f-e604-4b00-8bf4-14406b6c540c', '2b73d67f-3567-4b78-b83b-2eeebfb78d85', '3a414e49-f67f-4c5b-878c-2b3d0174d6cb', '4b752c60-30a4-4a5b-837a-74793a734f91', '67f17adf-ec65-488b-aa07-d3a0903c8237', 'add21b64-6362-49c7-a8fb-2b329e096d65', 'b0ac41bd-911f-49e3-990b-a210d9d40c05', 'e4248595-1f42-4524-942e-5ff7c23b607e', 'defcec85-cf01-4276-b18f-bad66ae75b03', '50f0d807-fec5-4019-b6fa-bc8f4cca9ea3', 'f52fefb1-d3b2-4695-b9e2-1bb364f1d39c', '0da95d26-926a-41d9-8177-16ed4d132558', 'd4f1e238-9d61-4b9d-87bb-0142e0649c49', 'e58253e1-3001-40f8-8146-dcab1041e37a', 'd5fbcb7b-e326-45d4-a0fd-aab6bf53f500', 'a9846286-2312-4926-8995-9727ec9e8f67', '2bd0b11a-1963-4727-9bc1-619fa01e6804', 'c849c72e-19b6-4f25-bd79-62b78a8a13f2', 'dfed45a5-0690-4b54-91cc-9db890ea898c', 'e9208b5e-48dc-41f6-a6a2-5a8a56e97c6c', 'f5ba694f-d760-4a09-ac92-853811e9ebb6', '101a7f88-3f94-4b0d-a4d4-2204ad2ddafa', '2fa116b6-da16-4302-bd00-0f413da5d24e', 'b0f9d6dc-f91a-4551-a6d2-fc85f74fbc73', '13ec8297-6f6a-46cd-b3d8-8c827e3e4a42', 'f102d6ee-2d9e-4aa0-b1f8-160e00727224', '7411a96b-f324-46ac-a607-9b57b0023e17', '4c61cd12-a7b1-4609-b6fd-43b39755c1f7', 'b84d0967-e2b2-46f6-9ad8-754f841f8c2b', 'cfff13ad-a7f3-4fa7-98a4-4d31b29d3b14', 'a7139a1b-9527-48b6-9585-4ec6e40bbe52', '2c582b5f-b5a2-47a1-9150-a278699efad5', '39506869-09f0-46cf-a50c-4d5ab1b40d51', '483585f4-a495-4e74-b7ca-27a5880df909', '6befa8b0-3d24-45bc-9f53-0dd89d070b6f', '6fa93fe5-4e80-4654-bf4c-c5dc53fe5dd5', '9bfbc7c5-5af1-4b23-b478-8453b053fce8', 'e7cd6d88-7397-40d0-aa61-33876854ec29', 'f2556bb8-8b8e-43c0-b960-5a00e434a52d', '2db837d9-9c67-487f-a18b-a04a7a348b71', 'e570b5c7-8d7e-4fff-9669-580df3f739dc', '4887a5d0-896d-47fa-9c0b-0247af55b7ae', '828d052b-3de5-4da9-ac29-d1682de1fbb0', 'dfa8519d-46d5-48af-a840-f712c5042aea', 'fe6f0ca1-9e2b-46c9-b52a-219138cb3170', '17ba8808-4ef7-4f28-9129-5a42f8f83e90', '08520138-46c0-451d-861a-41fc14cf65b6', '2d590b12-a3c8-4d85-9fe1-83ff9d83750f', '72dfbc00-46d6-4574-a8f9-be23b16df482', '84f7c68f-151b-4d68-a0fb-a87d6cbafcd6', '249ef90f-641c-4bf8-b8f8-a410c1fa591e', '0a8150ba-227b-4fa3-bc78-bcc5984d9abf', '13424f21-9677-4703-a80a-3e8f53fb3288', '1dd5c215-26f0-43c4-8a09-9336d7e886e6', '29fd0f9e-4877-4a5a-b6c8-f144d52405f4', '31eaf1fb-a4cf-4526-ab37-8d31b4fca07d', '428ce37c-f7e4-4a26-bfdb-dfb9c0bd42e0', '566b024c-0322-463d-8f5e-d249f779a460', 'b3142d4c-493b-4b40-bc80-d8d8d23eab74', '0e4f459a-03c2-4cf6-9ac4-fc88a1e31379', '7d197ab5-b31c-4fdd-8cd6-d1dcddd584c1', '9bbf9c30-31da-48af-891b-80dcb749af67', '1e5a5161-1985-4b4b-87a5-d949bdba76f2', '1ebae8d4-7efb-46d5-b06a-d9d1daa07a3b', '37190206-d9af-4e58-a9aa-ee9b9f850853', '4a2e47be-8ead-4e95-80e4-7643ae147ac8', 'a32e0afe-3577-4966-accc-064827164571', '2fb5c7c3-4f00-4e10-a45d-8d1fdca8aaf4', '4bd07b79-1481-4d09-a793-b96a1a18eac2', '53d070a5-3bdd-4010-9a1c-cc4861f27a7b', 'c6bbdfad-65a9-41d4-a4a3-66bb9d991a0e', 'f53e9828-4ff5-4d34-9644-8fdb3f469709', '09a1bd15-6aca-462b-8fc9-a46e0626f028', '2ddbd32c-0d59-473c-a42d-677a1173a9bb', '4dd19385-37ce-4aef-9d11-6422dc5d884c', '9ba16a8e-2548-4380-ac95-387b1759ccb1', 'd62745f5-53e3-422d-8c38-ad2a83a3b9be', '9125c17a-445b-4c8a-831e-57468f0f09a6', 'fdeca262-86fe-41db-bf8e-5384e6ac0c3c', '0880b4cc-0f06-451c-848a-20f3b103307a', '42cd2a3b-4b24-4bf5-8523-746d22182d2b', '59108572-1982-4bb3-985b-50f6745251d4', '80bc0e50-d3b3-4c3a-bb18-f2999a25a421', 'c14778f8-047f-4161-8e02-946ea7eda937', 'd8fd0126-fa29-447c-801c-cbe0939a450f', 'e9e2e0d6-9f8f-4493-9ab3-009eb9aa0621', '3ed2521f-a18e-4eaf-b31c-5dafe2130e07', 'd9cf21f1-6e7c-4cc3-a91c-e1337e0458f2', '5b93c4e4-c18e-4e44-9f32-aa58e743ea88', '89d60d6e-4012-49f4-a129-0c986b0d20ce', 'fe84b3d0-d22c-4ed8-afb0-55c0fb3218bd', '137fa172-16d7-4403-99a7-7fae524ad93d', '917f876c-e280-4e0f-a9fe-11871d09c332', 'ee0423ff-c782-469b-bbf5-7cb44c66a81e', '3108b0d2-fd65-4f1d-83c0-215c40dab3a7', '1bb00179-db42-45f8-94b0-9cef05a72753', '8704922d-33a2-438c-9ed3-6ca72b42c5fd', 'b330bd42-7325-4e42-8c73-6ec05e9d5d5c', 'e87ff15d-4395-45f9-be1f-af8e85e60039', '0c1cfb26-2d1c-4fe5-b1d0-7320af4a63f1', '7b7b562e-6f30-47ab-accb-d5fe49ca1bcd', '1ea49207-33e9-4ab9-8c0e-4636e0deccc4', '3394a5f6-df52-40f4-809a-e613945e25ad', '3e1f0bd3-5de0-4ee5-b8b9-d7e25702aaef', '5aa58e64-594f-467a-91c4-1f2d3e6218dc', '637971b1-21d8-4098-8889-4a22c473d145', '7b874cad-88fb-4c54-971c-c992c247d382', '92e19e72-8c3d-44f4-b0e7-e2f9273f1ab2', 'cd63a6ca-8824-4914-849e-450256e0036e', 'cdfaaa6a-1605-4fbc-86c8-79154b46c43f'], '@persistence_version': 1}
2023-01-24T08:07:58.566    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527868758):Number of requests made: 1; Number of events received: 2000; Number of duplicated events filtered out: 1683; Number of events generated and sent: 317; Average of events per second: 32.369.
2023-01-24T08:07:58.566    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> The data is up to date!
2023-01-24T08:07:58.566    INFO InputProcess::NebulaDnsLogsDataPuller(example_input,12345,dns_log_data,predefined) -> Data collection completed. Elapsed time: 9.808 seconds. Waiting for 590.192 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

Statistics for this pull cycle (@devo_pulling_id=1674492783536):Number of requests made: 1; Number of events received: 1; Number of duplicated events filtered out: 1; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-01-23T22:23:05.265    INFO InputProcess::NebulaNotificationsDataPuller(example_input,12345,suspicious_activity,predefined) -> The data is up to date!

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

...

Expand

title	Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Error Type	Error Id	Error Message	Cause	Solution
InitVariableError	1	`Invalid value is provided for the historic_date.`	historic_date_utc is mentioned in the wrong date time format.	Write the historic_date_utc in correct formatFor. ex. `2022-11-15T14:32:33Z`
InitVariableError	2	`Time format for historic date must be "%Y-%m-%dT%H:%M:%SZ"`	historic_date_utc is mentioned in the wrong date time format.	Write the historic_date_utc in correct formatFor. ex. `2022-11-15T14:32:33Z`
InitVariableError	3	`historic datetime cannot be greater than the present UTC time`	The historic datetime mentioned is of the future.	The value of historic datetime should always be lesser than current datetime
SetupError	100	`Error occurred while requesting access token from the Nebula server. Error message: .`	Not able to establish connection/ pull data due to incorrect credentials or account expired.	Re-check if the entered credentials are correct, and that the access token is expired or not
PullError	300	`Error occurred while retrieving data from Nebula server , Error details: .`	Invalid token or request	Re-check if the entered credentials are correct, and that the access token is expired or not
PullError	301	`"Error in the request query : ___ .`	Cause for this error will be you have entered invalid values in service parameters.	Check whether the parameters for that service are correct or not.
PullError	302	`Api key OR Username password are incorrect. status code: 401. Error message:`	Cause for this error will be you have entered invalid values for client_id, client_secret or account_id	Re-enter correct values for `client_id, client_secret or account_id`
PullError	303	`The resource requested is not found. Resource Url:`	Incorrect Url or endpoint	Re-enter `api_base_url and endpoint`.
PullError	304	`Something is wrong with the payload. Please provide the correct value in the config.yaml file.`	Enter invalid value in config.yaml file for that service.	Check whether the parameters for that service are correct or not.
PullError	305	`Unexpected error occurred at the Nebula server. status code:`	Cause for this error will be written in the Error message on the console.	As there are a number of causes for this error, check on the console as the reason is described and take action accordingly.

Collector operations

This section is intended to explain how to proceed with the specific operations of this collector.

Expand

title	Verify collector operations

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

Code Block

 INFO MainProcess::MainThread -> Added "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula" directory to the Python path
2023-02-22T10:11:12.878    INFO MainProcess::MainThread -> Added "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula/config_internal" directory to the Python path
2023-02-22T10:11:12.878    INFO MainProcess::MainThread -> Added "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula/schemas" directory to the Python path
2023-02-22T10:11:12.878    INFO MainProcess::MainThread -> Production mode: False, execute only setup and exit: False, Python version: "3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]", current dir: "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula", exists "config" dir: True, exists "config_internal" dir: True, exists "certs" dir: True, exists "schemas" dir: True, exists "credentials" dir: True
2023-02-22T10:11:12.900    INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-02-22T10:11:12.900    INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-02-22T10:11:12.901    INFO MainProcess::MainThread -> "/etc/devo/job" does not exists
2023-02-22T10:11:12.901    INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-02-22T10:11:12.901    INFO MainProcess::MainThread -> "/etc/devo/collector" does not exists
2023-02-22T10:11:12.901    INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "/home/metronlabs/Documents/Nebula2202/devo-MalwarebytesNebula/config/config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
2023-02-22T10:11:12.917    INFO MainProcess::MainThread -> Build time: "UNKNOWN", OS: "Linux-5.15.0-60-generic-x86_64-with-glibc2.29", collector(name:version): "example_collector:1.0.0", owner: "integrations_factory@devo.com", started at: "2023-02-22T04:41:12.909599Z"
2023-02-22T10:11:12.919    INFO MainProcess::MainThread -> Initialized all object from "MainProcess" process
2023-02-22T10:11:12.919    INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=120s)
2023-02-22T10:11:12.921    INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=120s)
2023-02-22T10:11:12.921    INFO OutputProcess::MainThread -> Process started
2023-02-22T10:11:12.924    INFO MainProcess::MainThread -> Started all object from "MainProcess" process

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Code Block

 INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-02-22T14:26:39.858    INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-02-22T14:26:39.858    INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Sender: DevoSender(standard_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2023-02-22T14:26:39.858    INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Standard - Total number of messages sent: 31, messages sent since "2023-02-22 08:51:39.785516+00:00": 31 (elapsed 0.009 seconds)
2023-02-22T14:26:39.864    INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-02-22T14:26:39.865    INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-02-22T14:26:39.865    INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> Sender: ConsoleSender(standard_senders,console_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
2023-02-22T14:26:39.865    INFO OutputProcess::ConsoleSenderManagerMonitor(standard_senders,console_1) -> Standard - Total number of messages sent: 31, messages sent since "2023-02-22 08:51:39.798224+00:00": 31 (elapsed 0.006 seconds)
2023-02-22T14:26:39.876    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-02-22T14:26:39.876    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-02-22T14:26:39.876    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2023-02-22T14:26:39.876    INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_eu_1) -> Lookup - Total number of messages sent: 0, messages sent since "2023-02-22 08:51:39.809055+00:00": 0 (elapsed 0.000 seconds)
2023-02-22T14:26:39.879    INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-02-22T14:26:39.879    INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-02-22T14:26:39.880    INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> Sender: ConsoleSender(lookup_senders,console_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
2023-02-22T14:26:39.880    INFO OutputProcess::ConsoleSenderManagerMonitor(lookup_senders,console_1) -> Lookup - Total number of messages sent: 0, messages sent since "2023-02-22 08:51:39.820290+00:00": 0 (elapsed 0.000 seconds)
2023-02-22T14:26:39.883    INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-02-22T14:26:39.883    INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-02-22T14:26:39.883    INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Sender: DevoSender(internal_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
2023-02-22T14:26:39.884    INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_eu_1) -> Internal - Total number of messages sent: 59, messages sent since "2023-02-22 08:51:39.830694+00:00": 59 (elapsed 0.578 seconds)
2023-02-22T14:26:39.884    INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-02-22T14:26:39.884    INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-02-22T14:26:39.884    INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> Sender: ConsoleSender(internal_senders,console_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
2023-02-22T14:26:39.884    INFO OutputProcess::ConsoleSenderManagerMonitor(internal_senders,console_1) -> Internal - Total number of messages sent: 59, messages sent since "2023-02-22 08:51:39.841277+00:00": 59 (elapsed 0.007 seconds)
2023-02-22T14:26:44.862    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Pull Started
2023-02-22T14:26:44.862    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> Consumed messages: 5, total_bytes: 3778 (60.00074 seconds)
2023-02-22T14:26:44.863    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:44.862", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"Pull Started\", \"time\": \"2023-02-22T08:56:44.862220Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"}
2023-02-22T14:26:44.863    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> Consumed messages: 5 messages (60.000787 seconds) => 0 msg/sec
2023-02-22T14:26:44.863    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Consumed messages: 5 messages (60.000704 seconds) => 0 msg/sec
2023-02-22T14:26:48.548    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/detections
2023-02-22T14:26:48.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Total detections available are = 31
2023-02-22T14:26:48.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Received 31 detections data from Nebula Server
2023-02-22T14:26:48.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Removing the duplicate detections if present...
2023-02-22T14:26:48.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Saved state: {'last_polled_timestamp': 1677056144.861205, 'historic_date_utc': None, 'ids_with_same_timestamp': ['0ab0b92f-4653-5f9a-870a-4c4aa128072d'], '@persistence_version': 1}
2023-02-22T14:26:48.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Received 0 detections data from Nebula Server after removing duplicates
2023-02-22T14:26:48.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Number of detections  sent to Devo: 0
2023-02-22T14:26:48.549    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-02-22T14:26:48.550    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-02-22T14:26:48.550    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-02-22T14:26:48.550    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> The data is up to date!
2023-02-22T14:26:48.551    INFO InputProcess::NebulaDetectionDataPuller(example_input,12345,detections,predefined) -> Data collection completed. Elapsed time: 3.689 seconds. Waiting for 56.311 second(s) until the next one
2023-02-22T14:26:48.553    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.550", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"(Partial) Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.\", \"time\": \"2023-02-22T08:56:48.550458Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"}
2023-02-22T14:26:48.553    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.550", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"Statistics for this pull cycle (@devo_pulling_id=1677056204862):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.\", \"time\": \"2023-02-22T08:56:48.550726Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"}
2023-02-22T14:26:48.553    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.550", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"The data is up to date!\", \"time\": \"2023-02-22T08:56:48.550900Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"}
2023-02-22T14:26:48.553    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:56:48.551", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"Data collection completed. Elapsed time: 3.689 seconds. Waiting for 56.311 second(s) until the next one\", \"time\": \"2023-02-22T08:56:48.551084Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null, \"input_name\": \"example_input\", \"service_name\": \"detections\", \"module_name\": \"NebulaDetectionDataPuller\"}"}
2023-02-22T14:27:39.966    INFO InputProcess::MainThread -> [GC] global: 32.8% -> 32.8%, process: RSS(45.85MiB -> 45.85MiB), VMS(503.38MiB -> 503.38MiB)
2023-02-22T14:27:39.967    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"timestamp": "2023-02-22 08:57:39.966", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"[GC] global: 32.8% -> 32.8%, process: RSS(45.85MiB -> 45.85MiB), VMS(503.38MiB -> 503.38MiB)\", \"time\": \"2023-02-22T08:57:39.966130Z\", \"level\": \"info\", \"collector_name\": \"example_collector\", \"collector_version\": \"1.0.0\", \"collector_image\": null}"}
2023-02-22T14:27:39.967    INFO InputProcess::MainThread -> global_status: {"input_process": {"process_id": 3653, "process_status": "running", "thread_counter": 7, "thread_names": ["MainThread", "QueueFeederThread", "NebulaDataPullerSetup(example_collector,example_input#12345,detections#predefined)", "QueueFeederThread", "NebulaDetectionDataPuller(example_input,12345,detections,predefined)", "ServiceThread(example_input,12345,detections,predefined)", "InputThread(example_input,12345)"], "memory_info": {"rss": "45.85MiB", "vms": "503.38MiB", "shared": "9.52MiB", "text": "2.42MiB", "lib": "0.00B", "data": "84.34MiB", "dirty": "0.00B"}, "input_threads": [[]], "running_flag": true, "message_queues": {"standard": {"name": "standard_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7fdc9e4b3df0>"}, "lookup": {"name": "lookup_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7fdc9e4e1f10>"}, "internal": {"name": "internal_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7fdc9e465700>"}}}}
2023-02-22T14:27:40.001    INFO OutputProcess::MainThread -> [GC] global: 32.8% -> 32.8%, process: RSS(43.34MiB -> 43.34MiB), VMS(1.55GiB -> 1.55GiB)

Info
By default, these information traces will be displayed every 10 minutes.

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

Info
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 31, messages sent since "2023-02-22 08:51:39.785516+00:00": 31 (elapsed 0.009 seconds)

Displays the number of events from the last time and following the given example, the following conclusions can be obtained:

31 events have been sent to Devo since the collector started.
The last checkpoint timestamp was "2023-02-22 08:51:39.785516+00:00": 31 .
31 events were sent to Devo between the last UTC checkpoint and now.
Those 31 events required 0.009 seconds to be delivered.

Info
By default, these traces will be shown every 10 minutes.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

Standard - Total number of messages sent: 31, messages sent since "2023-02-22 08:51:39.785516+00:00": 31 (elapsed 0.009 seconds)

Displays the number of events from the last time and following the given example, the following conclusions can be obtained:

31 events have been sent to Devo since the collector started.
The last checkpoint timestamp was "2023-02-22 08:51:39.785516+00:00": 31.
31 events were sent to Devo between the last UTC checkpoint and now.
Those 31 events required 0.009 seconds to be delivered.

...

Expand

title	Enable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Change log for v1.x.x

Release	Released on	Release type	Details	Recommendations
v1.0.0	12 May 2023	New collector	-	-

Page Comparison

Versions Compared

Old Version 29

New Version Current

Key

Overview

Devo collector features

Minimum configuration required for basic pulling

Overview

Devo collector features

Data sources

Vendor setup

Accepted authentication methods

Run the collector

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Collector services detail

Events service

Setup output

Puller output

Vulnerability management service

Setup output

Puller output

Notifications service

Setup output

Puller output

Suspicious Activity service

Setup output

Puller output

Detection service

Setup output

Puller output

DNS logs service

Setup output

Puller output

Collector operations

Initialization

Events delivery and Devo ingestion

Sender services

Sender statistics

Change log for v1.x.x