Page Comparison

Table of Contents

maxLevel	2
minLevel	2
type	flat

...

Here you find the necessary steps to enable the collection.

	Steps	Screenshots
1	Login to your Recorded Future account. Open the Menu in the upper-right corner and select the option User Settings.
2	Select the API Access tab. To create a new API token, click on Generate New API Token.
3	Enter a name for the token in the Name field. Then click on Generate.	-
4	Select Devo from the drop-down integration list	-
5	Click on the Generate new API token button. The new API token appears in the table below. Make a note of the token value as this is required for the ingest Configuration.	-

...

IpAddressLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Address_Threat_List
- my.lookuplist.Recorded_Future_IPv6_Address_Threat_List
FileHashLookupPuller:
- my.lookuplist.Recorded_Future_Domain_Threat_List
DomainLookupPuller:
- my.lookuplist.Recorded_Future_File_Hash_Threat_List
UrlLookupPuller:
- my.lookuplist.Recorded_Future_URL_Threat_List
VulnerabilityLookupPuller:
- my.lookuplist.Recorded_Future_Vulnerability_Threat_List
PublicUkraineRussiaIpsLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Public_Ukranie_Russia_List
- my.lookuplist.Recorded_Future_IPv6_Public_Ukranie_Russia_List

...

Lookup Puller Service

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> successfully generated new access token
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Terminated
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Setup for module <WizDataPuller><DataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Started.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
WARNING InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Terminated
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Started
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> 5 issues delivered
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Updating deduplication buffers content
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Terminated
INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

INFO InputProcess::WizDataPullerDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference. (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

...

Expand

title	Verify collector operations

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

Code Block

INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 1000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-eu.devo.io', 'port': 443, ...}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
INFO OutputProcess::MainThread -> Process started
INFO InputProcess::MainThread -> Process Started
INFO InputProcess::MainThread -> InitVariables Started
INFO InputProcess::MainThread -> Validating variables in collector definitions Started
INFO InputProcess::MainThread -> Flatten data is not provided in the config.yaml. Considering the flatten data from collector definitions
INFO InputProcess::MainThread -> Validating collector Variables is terminated
INFO InputProcess::MainThread -> Initialization of api_base_url has started.
INFO InputProcess::MainThread -> api_base_url has been initialized
INFO InputProcess::MainThread -> Initialization of credentials has started.
INFO InputProcess::MainThread -> credentials have been initialized.
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO InputProcess::MainThread -> InitVariables Terminated
INFO InputProcess::MainThread -> InputThread(wiz_data_puller,111) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> ServiceThread(wiz_data_puller,111,issues,predefined) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Starting thread
INFO InputProcess::MainThread -> WizDataPullerDataPuller(wiz_data_puller,111,issues,predefined) - Starting thread
WARNING InputProcess::WizDataPullerDataPuller(wiz_data_puller,111,issues,predefined) -> Waiting until setup will be executed
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> This is the first run of collector. Generating the access token
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Getting the auth token url based on provided api_base_url
INFO InputProcess::WizDataPullerSetupDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Using default Authentication Domain auth.wiz.io for fetching Access Token
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
INFO InputProcess::MainThread -> [GC] global: 36.7% -> 36.7%, process: RSS(26.93MiB -> 27.97MiB), VMS(334.43MiB -> 334.67MiB)
INFO OutputProcess::MainThread -> [GC] global: 36.7% -> 36.3%, process: RSS(26.68MiB -> 28.61MiB), VMS(910.71MiB -> 910.71MiB)
INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-eu.devo.io:443", ...}

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Code Block

INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)

Info
By default, these information traces will be displayed every 10 minutes.

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services	Description
`internal_senders`	In charge of delivering internal metrics to Devo such as logging traces or metrics.
`standard_senders`	In charge of delivering pulled events to Devo.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

Info
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.

Info
By default these traces will be shown every 10 minutes.

...

Version	Old Version 13	New Version 14
Changes made by	Former user	Former user
Saved on	Sept 13, 2022	Sept 13, 2022

Versions Compared

Key

Lookup Puller Service

Setup output

Puller output

Initialization

Events delivery and Devo ingestion

Sender services

Sender statistics