Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: Code Block |
---|
INFO InputProcess::MainThread -> NebulaEventsDataPuller(example_input,12345,events,predefined) - Starting thread
2023-01-23T16:16:31.386 WARNING InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Waiting until setup will be executed
2023-01-23T16:16:31.386 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Token has expired. Generating the new one
2023-01-23T16:16:31.387 WARNING InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> The token/header/authentication is expired and it needs to be refreshed
2023-01-23T16:16:31.388 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Requesting access token from the Nebula server
2023-01-23T16:16:31.402 INFO OutputProcess::MainThread -> [GC] global: 25.0% -> 25.0%, process: RSS(46.83MiB -> 47.60MiB), VMS(1.19GiB -> 1.19GiB)
2023-01-23T16:16:31.408 INFO InputProcess::MainThread -> [GC] global: 25.0% -> 25.0%, process: RSS(46.96MiB -> 47.29MiB), VMS(791.23MiB -> 791.48MiB)
2023-01-23T16:16:31.720 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140563744962544"
2023-01-23T16:16:31.721 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"url": "collector-eu.devo.io:443", "chain_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/chain.crt", "cert_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.crt", "key_path": "/home/metronlads/Documents/nebula_bug/devo-MalwarebytesNebula/certs/if_metronlabs.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "metronlabs", session_id: "140563744962400"
2023-01-23T16:16:32.343 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Requesting access token from the Nebula server
2023-01-23T16:16:32.344 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Successfully generated new access token. Token is valid till: 2023-01-23 16:46:31
2023-01-23T16:16:32.344 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Previously generated token is still valid. Skipping the generation of new access token
2023-01-23T16:16:32.344 INFO InputProcess::NebulaDataPullerSetup(example_collector,example_input#12345,events#predefined) -> Setup for module <NebulaEventsDataPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: Info |
---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
Code Block |
---|
INFO MainThread -> [INPUT] ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined023-01-24T08:03:26.575 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Pull Started
2023-01-24T08:03:27.586 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> WaitingResponse untilreceived setupfrom willNebula beserver executedResource INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> PrePull Started
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> PrePull terminated
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Starting data collection every 5 seconds
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Pull Started. Retrieving timestamp: 2022-06-28 13:00:59.276966+00:00
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started getting events from ThreatQuotient
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started getting events from ThreatQuotient
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started sending events to Devo
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656421259.276966): Number of requests performed: 2; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 2 (from 1 unflattened events); Average of events per second: 4.179186813829765.
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> last_fetched_event_id and last_update_time saved in state: {'last_polled_time': 1656342870.739774, 'reset_persistence_auth': '', 'all_events_ids': [17653]}
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Last polled time saved in state: {'last_polled_time': 1656421259.276966, 'reset_persistence_auth': '', 'all_events_ids': [17653]}
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Pull terminated
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefinedUrl: https://api.malwarebytes.com/nebula/v1/events?start=2023-01-24T02:32:26Z
2023-01-24T08:03:27.588 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Removing the duplicate events if present...
2023-01-24T08:03:27.589 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Number of events sent to Devo: 0
2023-01-24T08:03:27.589 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Total number of events: 0
2023-01-24T08:03:27.590 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> State last_polled_timestamp is updated with retrieving timestamp
2023-01-24T08:03:27.591 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Saved state: {'last_polled_timestamp': 1674527606.575356, 'historic_date_utc': None, 'ids_with_same_timestamp': ['0fa33de2-963a-4b7f-b709-4111eb82712c'], '@persistence_version': 1}
2023-01-24T08:03:27.591 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1674527606575):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-01-24T08:03:27.593 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> The data is up to date!
2023-01-24T08:03:27.595 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Data collection completed. Elapsed time: 01.483019 seconds. Waiting for 458.517980 second(s) until the next one
|
After a successful collector’s execution (that is, no error logs found), you will see the following log message: Code Block |
---|
2023-01-24T08:03:27.591 INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefinedInputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1655983326.2908481674527606575): Number of requests performedmade: 21; Number of events received: 520; Number of duplicated events filtered out: 0; Number of events generated and sent: 52 (from 52 unflattened events)0; Average of events per second: 92.99414315733.0.000.
2023-01-24T08:03:27.593 INFO InputProcess::NebulaEventsDataPuller(example_input,12345,events,predefined) -> The data is up to date! |
Info |
---|
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window. |
|