Page Comparison

Table of Contents

maxLevel	2
minLevel	2
type	flat

...

Feature	Details
Allow parallel downloading (`multipod`)	`Not allowed`
Running environments	`Collector server` `On-premise`
Populated Devo events	`Lookups`
Flattening Preprocessing	`No`

Data sources

Steps

Screenshots

1

Data Source	Description	API Endpoint	Collector service name	Devo Table	Available from release
IpAddressLookupPuller	This endpoint provides a list of IPs classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/ip/risklist?format=csv%2Fsplunk`	`lookup_puller` type: `ip`	Lookup `my.lookuplist.Recorded_Future_IPv4_Address_Threat_List` `my.lookuplist.Recorded_Future_IPv6_Address_Threat_List`	`v1.0.0`
DomainLookupPuller	This endpoint provides a list of domains classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/domain/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: domain`	Lookup `my.lookuplist.Recorded_Future_Domain_Threat_List`	`v1.0.0`
FileHashLookupPuller	This endpoint returns a list of file hashes classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/hash/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: hash`	Lookup `my.lookuplist.Recorded_Future_File_Hash_Threat_List`	`v1.0.0`
UrlLookupPuller	This endpoint returns a list of URL classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/url/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: url`	Lookup `my.lookuplist.Recorded_Future_URL_Threat_List`	`v1.0.0`
VulnerabilityLookupPuller	This endpoint returns a list of vulnerabilities classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/vulnerability/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: vulnerability`	Lookup `my.lookuplist.Recorded_Future_Vulnerability_Threat_List`	`v1.0.0`
PublicUkraineRussiaIpsLookupPuller	This endpoint returns a list of IPs related with Russia and Ukraine.	`https://api.recordedfuture.com/v2/fusion/files/?path=/public/ukraine/ukraine_russia_ip.csv`	`lookup_puller` `type: PublicUkraineRussiaIps`	Lookup `my.lookuplist.Recorded_Future_IPv4_Public_Ukranie_Russia_List` `my.lookuplist.Recorded_Future_IPv6_Public_Ukranie_Russia_List`	`v1.2.0`

Vendor setup

This collector only requires a source configuration by generating a Recorded Future API key. Recorded Future clients with Advanced or Core licenses, and one or more Connect API subscriptions can create and manage their API tokens directly in the Recorded Future portal.

Enable the collection

Here you find the necessary steps to enable the collection.

List

v1.2.0

Vendor setup

There are some minimal requirements to enable this collector:

Login on Recorded Future
Create a new token

Action	Steps
Login on Recorded Future	Login to your Recorded Future account

. Open the Menu in the upper-right corner

at app.recordedfuture.com
Create a new token	Click on Menu and select the option User Settings.

Image Removed

2

Select the API Access tab.

To

To create a new API token, click on Generate New API Token.

Image Removed3

Enter a name for the token

in the Name field. Then click on Generate

.

-

4

Select Devo from the

drop-down

integration list.

-

5

Click on the Generate new API token button.

The new API token appears in the table below.

Make a note of the token value, as this is required for the

ingest

Ingest Configuration.

-

Minimum configuration required for basic pulling

...

This section is intended to explain how to proceed with specific actions for services.

Lookup

...

puller service

The only service available in this collector is lookup_puller. It contains 6 different pullers:

IpAddressLookupPuller
FileHashLookupPuller
DomainLookupPuller
UrlLookupPuller
VulnerabilityLookupPuller
PublicUkraineRussiaIpsLookupPuller

Expand

title	Devo categorization and destination

All events of this service are ingested into these tables:

IpAddressLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Address_Threat_List
- my.lookuplist.Recorded_Future_IPv6_Address_Threat_List
FileHashLookupPuller:
- my.lookuplist.Recorded_Future_Domain_Threat_List
DomainLookupPuller:
- my.lookuplist.Recorded_Future_File_Hash_Threat_List
UrlLookupPuller:
- my.lookuplist.Recorded_Future_URL_Threat_List
VulnerabilityLookupPuller:
- my.lookuplist.Recorded_Future_Vulnerability_Threat_List
PublicUkraineRussiaIpsLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Public_Ukranie_Russia_List
- my.lookuplist.Recorded_Future_IPv6_Public_Ukranie_Russia_List

...

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> successfully generated new access token
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from  server
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Terminated
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Setup for module <DataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> PrePull Started.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
WARNING InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> PrePull Terminated
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Started
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting  API for issues
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from 
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting  API for issues
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from 
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting  API for issues
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from 
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 5 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Updating deduplication buffers content
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Terminated
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference. (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

...

Expand

title	Enable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Change log for v1.x.x

...

Release

...

Released on

...

Release type

...

Details

...

Recommendations

...

v1.0.0

...

20 May 2021

...

Status

colour	Green
title	FEATURES

...

New features:

Initial release with the following Recorded Future Threat List default data sources:
- IPs
- Domains
- URLs
- File Hashes
- Vulnerabilities

...

Upgrade

...

v1.1.0

...

deployment mode.

Change log for v1.x.x

Release

Released on

Release type

Details

Recommendations

v1.0.2

20 May 2021

Status

colour

PurpleUpgrade

title
Green

IMPROVEMENTS

Status

colour	Yellow
title	VULNS

Improvement:

Upgrade the IFC SDK Lookup Factory Service to improve the data model validation.
The underlay Devo Collector SDK has been upgraded to v1.1.4 to improve efficiency, increase the resilience and mitigate vulnerabilities.
The hard-reset procedure when losing connection with Devo has been improved.

Vulnerabilities mitigated:

CVE-2022-1664
CVE-2021-33574
CVE-2022-23218
CVE-2022-23219
CVE-2019-8457
CVE-2022-1586
CVE-2022-1292
CVE-2022-2068
CVE-2022-1304
CVE-2022-1271
CVE-2021-3999
CVE-2021-33560
CVE-2022-29460
CVE-2022-29458
CVE-2022-0778
CVE-2022-2097
CVE-2020-16156
CVE-2018-25032

FEATURES

Initial release with the following Recorded Future Threat List default data sources:

IPs
Domains
URLs
File Hashes
Vulnerabilities

Upgrade to v1.2.0

v1.1.0

24 May 2022

Status

colour	Green
title	FEATURES

Status

colour	Yellow
title	VULNERABILITIES

This release includes:

Upgrade the base docker-image from Debian to Ubuntu20 for vulnerabilities mitigation.
Upgrade the IFC SDK Lookup Factory Service to improve the data model validation.

Upgrade to v1.2.0

v1.2.0-stable

01 Jun 2022

Status

colour	Green
title	FEATURES

New features:

Added new custom data

source for retrieving

sources from Recorded Future Threat List service:

IPs related to Ukraine and Russia countries.

Recommended version

Configuration checklist

Here you will find a brief checklist of the important configurations that need to be done for deploying this collector:

...

Version	Old Version 14	New Version 15
Changes made by	Former user	Former user
Saved on	Sept 13, 2022	Sept 14, 2022

Versions Compared

Key

Data sources

Vendor setup

Enable the collection

Vendor setup

Minimum configuration required for basic pulling

Lookup

puller service

Setup output

Puller output

Change log for v1.x.x

Change log for v1.x.x

Configuration checklist