Page Comparison

...

Expand

title	Devo categorization and destination

All events of Vulnerability service are ingested into the table my.app.nebula.vulnerability_management.

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> InputThread(example_input,12345) - Starting thread (execution_period=60s)
2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> ServiceThread(example_input,12345,vulnerability_management,predefined) - Starting thread (execution_period=60s)
2023-01-23T17:09:18.002    INFO InputProcess::MainThread -> NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Starting thread
2023-01-23T17:09:18.003    INFO InputProcess::MainThread -> NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) - Starting thread
2023-01-23T17:09:18.003    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Token has expired. Generating the new one
2023-01-23T17:09:18.004 WARNING InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Waiting until setup will be executed
2023-01-23T17:09:18.004 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> The token/header/authentication is expired and it needs to be refreshed
2023-01-23T17:09:18.005    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server
2023-01-23T17:09:18.020    INFO OutputProcess::MainThread -> [GC] global: 25.8% -> 25.9%, process: RSS(46.42MiB -> 48.71MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-23T17:09:18.029    INFO InputProcess::MainThread -> [GC] global: 25.9% -> 25.9%, process: RSS(47.31MiB -> 47.38MiB), VMS(791.48MiB -> 791.48MiB)
2023-01-23T17:09:18.341    INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332628086400"
2023-01-23T17:09:18.344    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140332642608512"
2023-01-23T17:09:19.010    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Requesting access token from the Nebula server
2023-01-23T17:09:19.011    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 17:39:18
2023-01-23T17:09:19.012    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Previously generated token is still valid. Skipping the generation of new access token 
2023-01-23T17:09:19.012    INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,vulnerability_management#predefined) -> Setup for module <NebulaVulnerabilityDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

2023-01-23T17:19:40.513    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Pull Started
2023-01-23T17:19:41.573    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/export
2023-01-23T17:19:41.574    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Received 5 CVE ids from Nebula Server
2023-01-23T17:19:41.575    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Removing the duplicate cve if present...
2023-01-23T17:19:41.575    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2022-34716'}
2023-01-23T17:19:42.498    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2022-34716
2023-01-23T17:19:42.499    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2022-24464'}
2023-01-23T17:19:43.419    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2022-24464
2023-01-23T17:19:43.419    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2020-8927'}
2023-01-23T17:19:44.393    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2020-8927
2023-01-23T17:19:44.395    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2021-34485'}
2023-01-23T17:19:45.339    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2021-34485
2023-01-23T17:19:45.341    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Fetching information for particular id = {'id': 'CVE-2021-26423'}
2023-01-23T17:19:46.356    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Response received from Nebula server Resource Url: https://api.malwarebytes.com/nebula/v1/cve/CVE-2021-26423
2023-01-23T17:19:46.359    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Number of vulnerabilities sent to Devo: 5
2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Saved state: {'last_polled_timestamp': 1674474580.484891, 'historic_date_utc': 1669991553.0, 'ids_with_same_timestamp': ['CVE-2021-26423'], '@persistence_version': 1}
2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674474580484):Number of requests made: 1; Number of events received: 5; Number of duplicated events filtered out: 0; Number of events generated and sent: 5; Average of events per second: 0.855.
2023-01-23T17:19:46.362    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> The data is up to date!
2023-01-23T17:19:46.363    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Data collection completed. Elapsed time: 5.879 seconds. Waiting for 594.121 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

2023-01-23T17:19:46.361    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674474580484):Number of requests made: 1; Number of events received: 5; Number of duplicated events filtered out: 0; Number of events generated and sent: 5; Average of events per second: 0.855.
2023-01-23T17:19:46.362    INFO InputProcess::NebulaVulnerabilityDataPuller(example_input,12345,vulnerability_management,predefined) -> The data is up to date!

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Expand

title	Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Error type	Error ID	Error message	Cause	Solution
SetupError	1	`api_base_url must be specified`	The `api_base_url` setting is missing.	Make sure the `api_base_url` setting is present under the `events` service in your configuration.
	2	`api_base_url not of expected type: str`	The `api_base_url` setting has a type other than string.	Make sure the `api_base_url` setting is a string.
	3	`api_base_url must match one of two regexes: [...]`	The `api_base_url` setting does not follow the expected format.	Make sure your `api_base_url` has this format: `http[s]://{ip_address_or_domain}{:optional_port}`
	4	`Required setting, credentials not found in user configuration`	There is no `credentials` section in your input settings.	Make sure there is a `credentials` section under the `threatquotient_data_puller` input in your configuration.
	5	`Required setting, credentials not of expected type: dict`	The `credentials` section is empty or has a simple type (is not an object).	Make sure the `credentials` section has a `username` and `password` fields.
	6	`Required setting, username not found in user configuration`	The `username` setting is missing.	Make sure the `username` setting from the `credentials` section has a value.
	7	`Required setting, username not of expected type: str`	The `username` setting has a type other than string.	Make sure the `username` setting from the `credentials` section is a string.
	8	`Required setting, password not found in user configuration`	The `password` setting is missing.	Make sure the `password` setting from the `credentials` section has a value.
	9	`Required setting, password not of expected type: str`	The `password` setting has a type other than string.	Make sure the `password` setting from the `credentials` section is a string.
	10	`Optional setting, verify_host_ssl_cert not of expected type: bool`	The `verify_host_ssl_cert` setting has a type other than boolean.	Make sure the `verify_host_ssl_cert` setting is a boolean value (`true`/`false`).
	11	`event_fetch_limit_in_items must be greater than or equal to [...] and less than equal to [...]`	The `event_fetch_limit_in_items` setting has a value too low or too high for the specified limits.	Make sure the `event_fetch_limit_in_items` setting is an integer ranged between the specified limits.
	12	`devo_tag_map must have an entry named "default"`	This error is not expected to happen in a regular flow.	This needs to be troubleshooted by the colllector’s developers.
	13	`Required setting, reset_persistence_auth not of expected type: str`	The `reset_persistence_auth` setting has a value, but its type is other than string.	Make sure the `reset_persistence_auth` setting is a string.
	14	`Required setting, historical_poll_datetime not of expected type: str`	The `historical_poll_datetime` setting has a type other than string.	Make sure the `historical_poll_datetime` setting is a string.
	15	`historical_poll_datetime does not match expected format [...]`	The `historical_poll_datetime` setting does not look like a valid date.	Make sure the `historical_poll_datetime` setting meets the mentioned format (a reference of this representation can be found here).
	16	`Please enter valid date for historical_poll_datetime less than or equal to one year`	The `historical_poll_datetime` setting is a date older than one year.	Make sure the `historical_poll_datetime` setting does not represent a date older than one year.
	17	`Please enter valid date for historical_poll_datetime less than or equal to the current date`	The `historical_poll_datetime` setting is a future date.	Make sure the `historical_poll_datetime` setting does not represent a future date.
InitVariablesError	100	`Unexpected status code when fetching ThreatQuotient JWT: [...]`	When a token was retrieved, the response had an unexpected error code.	Make sure your credentials are correct.
	101	`Unexpected status code when fetching ThreatQuotient client_id: [...]`	The collector is having issues connecting to the ThreatQ instance.	Make sure you have properly configured the `api_base_url` setting and that you can access the `{api_base_url}/assets/js/config.js` URL.
	102	`Cannot parse client_id from ThreatQuotient server`	The collector was expecting to find the Client’s ID, but could not find it. This is likely because the ThreatQ has been upgraded and the collector does not support it.	This needs to be troubleshooted by the colllector’s developers.
ApiError	400	`Unexpected status code when fetching ThreatQuotient events: [...]`	This error happens when the collector tries to fetch the ThreatQ events from its REST API.	In this error you will find the HTTP error code as long as the response’s text. This information should be enough to understand why is the error happening. Otherwise, please contact support.

...

Versions Compared

Old Version 20

New Version 21

Key

Setup output

Puller output