Page Comparison

Table of Contents

maxLevel	2
minLevel	2
type	flat

Overview

This collector ingests Recorded Future Threat Intelligence data into Devo as lookup tables to enable the following use cases:

Alerting - detects and alerts on potential security threats through correlation with other data types ingested into Devo, for example, firewall, proxy, or EDR logs.
Alert enrichment - adds contextual data about each entity and enriches security alerts.

The collector is available to all Devo customers with a valid Recorded Future license and API subscription.

Benefits

Reduce dwell time - through the correlation of Recorded Future Threat Intelligence and machine-generated data from systems in the network, your Security team can uncover threats that they would otherwise not know about and therefore reduce the dwell time of potential cyber-attacks.
Reduce mean-time-to-respond (MTTR) - enriched alerts provide your Security team with additional context about detected threats. This context can help to reduce the time required to complete the triage and /or investigation of the alert and inform a suitable response to mitigate the threat.

Integration architecture

...

Configuration

Recorded Future configuration

The only source configuration required is Recorded Future is a threat intelligence provider that allows you to access known bad incidents of compromise and entity enrichment capabilities. It has 6 different modules and charges on a per-user basis for access to the product. The 6 different modules are:

Security Operations: Providing intel into SIEM / SOAR platforms.
Standalone Threat Intelligence: An extension of Security Operations providing context and enrichment of known and emerging threats/incidents.
Brand Intelligence: Monitoring an organization’s external exposure.
Vulnerability Management: Intelligence into the prioritization of threats.
Third party intelligence: Data from third party sources.
Geo-political: More focussed on nation-state attacks and threat indicators.

Note
Recorded Future also charges customers for each integration they use. For example, a mutual customer of Recorded Future and Devo using this integration will pay Recorded Future a subscription fee.

Devo collector features

Feature	Details
Allow parallel downloading (`multipod`)	`Not allowed`
Running environments	`Collector server` `On-premise`
Populated Devo events	`Lookups`

Data sources

Data Source	Description	API Endpoint	Collector service name	Devo Table	Available from release
IpAddressLookupPuller	This endpoint provides a list of IPs classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/ip/risklist?format=csv%2Fsplunk`	`lookup_puller` type: `ip`	Lookup `my.lookuplist.Recorded_Future_IPv4_Address_Threat_List` `my.lookuplist.Recorded_Future_IPv6_Address_Threat_List`	`v1.0.0`
DomainLookupPuller	This endpoint provides a list of domains classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/domain/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: domain`	Lookup `my.lookuplist.Recorded_Future_Domain_Threat_List`	`v1.0.0`
FileHashLookupPuller	This endpoint returns a list of file hashes classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/hash/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: hash`	Lookup `my.lookuplist.Recorded_Future_File_Hash_Threat_List`	`v1.0.0`
UrlLookupPuller	This endpoint returns a list of URL classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/url/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: url`	Lookup `my.lookuplist.Recorded_Future_URL_Threat_List` ???	`v1.0.0`
VulnerabilityLookupPuller	This endpoint returns a list of vulnerabilities classified as Threat by Recorded Future.	`https://api.recordedfuture.com/v2/vulnerability/risklist?format=csv%2Fsplunk`	`lookup_puller` `type: vulnerability`	Lookup `my.lookuplist.Recorded_Future_Vulnerability_Threat_List`	`v1.0.0`
PublicUkraineRussiaIpsLookupPuller	This endpoint returns a list of IPs related with Russia and Ukraine.	`https://api.recordedfuture.com/v2/fusion/files/?path=/public/ukraine/ukraine_russia_ip.csv`	`lookup_puller` `type: PublicUkraineRussiaIps`	Lookup `my.lookuplist.Recorded_Future_IPv4_Public_Ukranie_Russia_List` `my.lookuplist.Recorded_Future_IPv6_Public_Ukranie_Russia_List`	`v1.2.0`

Vendor setup

This collector only requires a source configuration by generating a Recorded Future API key. Recorded Future clients with Advanced or Core licenses, and one or more Connect API subscriptions can create and manage their API tokens directly in the Recorded Future portal.

Enable the collection

Here you find the necessary steps to enable the collection.

...

1

Steps

Screenshots

2

Login to your Recorded Future account

...

. Open the Menu in the upper-right corner

...

and select the option User Settings

...

.	Image Added

...

3	Select the API Access tab.

...

To create a new API token, click on Generate New API Token.

...

Image Added

...

4	Enter a name for the token in the Name field. Then click on Generate.	-
5	Select Devo from the drop-down integration list

...

	-
6	Click on the Generate new API token button. The new API token appears in the table below. Make a note of the token value as this is required for the ingest

...

Configuration.

Devo configuration

The integration is hosted by Devo, enabling cloud-to-cloud ingestion of data. To enable the integration in your Devo domain:

Contact Devo support and provide your Recorded Future API token.
Devo support will then enable the integration on your behalf.

Using the integration

View lookup tables

To view the Devo lookup tables created by the integration:

Log in to your Devo domain.
Open the Data Search menu and click the Lookup Management tab.
You will see the new lookup tables in the Lookup List table, where you can view the number of rows included in the lookup and the time that the lookup table was last updated.
Image Removed
Hover over the right-hand side of a row in the Lookup List table to view summary information and manage the lookup table.

Run some sample correlation queries

With the lookup tables installed, you can use the data to run some sample queries. The examples below are designed to give you a starting point with the new data.

...

Rw step

Log in to your Devo domain.

Rw step

Open the Data Search menu.

Rw step

In the Explore Your Data tab, click the Free Text Query sub-tab.

Rw step

Copy and paste one of the queries from the table below to view the results:

...

Lookup

...

Query description

...

Query

...

IPv4 Address Threat List

...

Browse the data included in the IPv4 Threat List

...

Code Block
from my.lookuplist.Recorded_Future_IPv4_Address_Threat_List

...

View the srcIP address of devices attempting to connect to malicious IP addresses

...

Code Block

from firewall.all.traffic
select `lu/Recorded_Future_IPv4_Address_Threat_List/Risk`(dstIp)
as RecordedFutureRiskScore
select `lu/Recorded_Future_IPv4_Address_Threat_List/RiskString`(dstIp)
as RecordedFutureRiskString
select `lu/Recorded_Future_IPv4_Address_Threat_List/EvidenceDetails`(dstIp)
as RecordedFutureEvidence
where isnotnull(RecordedFutureRiskScore)
group dstIp, srcIp
select count(srcIp) as Count

...

View the users attempting to connect to malicious IP addresses

...

Code Block

from proxy.all.access
select `lu/Recorded_Future_IPv4_Address_Threat_List/Risk`(dstIp)
as RecordedFutureRiskScore
select `lu/Recorded_Future_IPv4_Address_Threat_List/RiskString`(dstIp)
as RecordedFutureRiskString
select `lu/Recorded_Future_IPv4_Address_Threat_List/EvidenceDetails`(dstIp)
as RecordedFutureEvidence
where isnotnull(RecordedFutureRiskScore)
group dstIp, dstHost, user
select count(user) as Count

...

Domain Threat List

...

Browse the data included in the domain threat list

...

Code Block
from my.lookuplist.Recorded_Future_Domain_Threat_List

...

View the users attempting to connect to potentially malicious domains and how many times

...

Code Block
from proxy.all.access select `lu/Recorded_Future_Domain_Threat_List/Risk`(dstHost) as RecordedFutureRiskScore where isnotnull(RecordedFutureRiskScore) group dstHost, user select count(user) as Count

...

File Hash Threat List

...

Browse the data included in the file hash threat list

...

Code Block
from my.lookuplist.Recorded_Future_File_Hash_Threat_List

...

Correlates Recorded Future File Hash intelligence with the sha256hash field in the edr.all.threats union table

...

Code Block

from edr.all.threats
select `lu/Recorded_Future_File_Hash_Threat_List/Risk`(sha256hash)
as RecordedFutureRiskScore
select `lu/Recorded_Future_File_Hash_Threat_List/RiskString`(sha256hash)
as RecordedFutureRiskString
select `lu/Recorded_Future_File_Hash_Threat_List/EvidenceDetails`(sha256hash)
as RecordedFutureEvidence
where isnotnull(RecordedFutureRiskScore)

Install some sample alerts

A primary use case for ingesting threat intelligence into Devo is to drive alerting to detect potential threats in your environment. The examples below are designed to give you a starting point to create alerts based on Recorded Future Threat Intelligence.

Connection to potentially malicious IPv4 Address (Firewall)

Note
This alert requires the `firewall.all.traffic` table in your Devo domain.

...

Rw step

Log in to your Devo domain.

Rw step

Open the Data Search menu.

Rw step

In the Explore Your Data tab, click the Free Text Query sub-tab.

Rw step

Copy and paste the query below and click the Run button.

Code Block

from firewall.all.traffic
select `lu/Recorded_Future_IPv4_Address_Threat_List/Risk`(dstIp) as RecordedFutureRiskScore
select `lu/Recorded_Future_IPv4_Address_Threat_List/RiskString`(dstIp) as RecordedFutureRiskString
select `lu/Recorded_Future_IPv4_Address_Threat_List/EvidenceDetails`(dstIp) as RecordedFutureEvidence
where isnotnull(RecordedFutureRiskScore)
group every 1m by source, fwname, action, srcIp, srcZone, dstIp, dstZone, RecordedFutureRiskScore, RecordedFutureRiskString, RecordedFutureEvidence

Rw step

From the search window, click the Alert definition button.

Rw step

Complete the New Alert Definition form using the values in the table below and click Create.

...

Summary

...

Connection to potentially malicious IPv4 Address from $srcIp

...

Description

...

A connection to a potentially malicious IP address was detected on $eventdate from $srcIp to $dstIp using the $source firewall.

Log Source: $source

Firewall: $fwname

Action: $action

Source: $srcIp ($srcZone)

Destination: $dstIp ($dstZone)

Recorded Future Risk Score: $RecordedFutureRiskScore

Recorded Future Risk Rules: $RecordedFutureRiskString

Recorded Future Evidence: $RecordedFutureEvidence

...

Alert Name

...

Connection to potentially malicious IPv4 Address - Firewall

...

Sub-category

...

Recorded Future Intel

...

Recommended Priority

...

Medium

...

Alert Type

...

Each

The new alert is created and can be viewed in the Administration → Alert Configuration menu.

Connection to a potentially malicious domain

Note
This alert requires the `proxy.all.access` table in your Devo domain.

...

Rw step

Log in to your Devo domain.

Rw step

Open the Data Search menu.

Rw step

In the Explore Your Data tab, click the Free Text Query sub-tab.

Rw step

Copy and paste the query below and click the Run button.

Code Block

from proxy.all.access
select `lu/Recorded_Future_Domain_Threat_List/Risk`(dstHost) as RecordedFutureRiskScore
select `lu/Recorded_Future_Domain_Threat_List/RiskString`(dstHost) as RecordedFutureRiskString
select `lu/Recorded_Future_Domain_Threat_List/EvidenceDetails`(dstHost) as RecordedFutureEvidence
where isnotnull(RecordedFutureRiskScore)
group every 1m by source, user, location, srcIp, srcHost, dstIp, dstHost, hitMiss_requestStat, RecordedFutureRiskScore, RecordedFutureRiskString, RecordedFutureEvidence

Rw step

From the search window, click the Alert definition button.

Rw step

Complete the New Alert Definition form using the values in the table below and click Create.

...

Summary

...

A connection to a potentially malicious domain was detected by $user

...

Description

...

A connection to a potentially malicious domain ($dstHost) was detected on $eventdate by $user using the $source proxy.

Source: $source

User: $user

Source: $srcIp (Host: $srcHost, Location: $location)

Destination: $dstIp ($dstHost)

Action: $hitMiss_requestStat

Recorded Future Risk Score: $RecordedFutureRiskScore

Recorded Future Risk Rules: $RecordedFutureRiskString

Recorded Future Evidence: $RecordedFutureEvidence

...

Alert Name

...

Connection to a potentially malicious domain

...

Sub-category

...

Recorded Future Intel

...

Recommended Priority

...

Medium

...

Alert Type

...

Each

...

-

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

Info
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details
`url_value`	This param refers to the endpoint used by the collector to pull data.
`api_token_value`	This is the access token provided by Recorded Future.
`list_of_sources`	This configuration allows you to define what data sources will be pulled.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Rw ui tabs macro

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Image Added

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

globals:
  debug: <debug_status>
  id: <collector_id>
  name: <collector_name>
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  recorded_future:
    id: <input_id>
    enabled: <input_status>
    requests_per_second: <requests_per_second>
    base_url: <base_url>
    api_token: <api_token_value>
    services:
      lookup_puller:
        # Intelligence Types Not Required Can Be Removed / Commented Out
        types:
          - <type_1>
          - <type_2>
          - ...
          - <type_N>

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Replace the placeholders with your required values following the description table below:

Parameter

Data Type

Type

Value Range

Details

debug_status

bool

Mandatory

false / true

If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.

collector_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique ID to this collector.

collector_name

str

Mandatory

Minimum length: 1
Maximum length: 10

Use this param to give a valid name to this collector.

devo_address

str

Mandatory

collector-us.devo.io
collector-eu.devo.io

Use this param to identify the Devo Cloud where the events will be sent.

chain_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt

cert_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.cert downloaded from your Devo domain.

key_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.key downloaded from your Devo domain.

input_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

If the value is true, the input definition will be executed. If the value is false, the service will be ignored.

requests_per_second

int

Optional

Minimum value: 1

Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.

Info
This parameter should be removed if it is not used.

base_url

str (with placesholders)

Mandatory

The URL must comply with the following format:

https://api.recordedfuture.com/v2/{0}/risklist?format=csv/splunk

Use this param to define the URL used by the collector to pull data.

Take into account that {0} is a placeholder for data type.

Info
This parameter is used for `ip`, `hash`, `domain`, `url` and `vulnerability` data sources.

api_token_value

str

Mandatory

Minimum length: 1

Set up here your access token provided by Recorded Future.

type_X

str (types is a `list` of `str` in yaml format)

Mandatory

Minimum data sources: 1

This configuration allows you to define what data sources will be pulled.

Info

Keep in mind that each data source has different refresh times associated with it:

ip → 1 hour
hash → 1 day
domain -> 2 hours
url -> 1 day
vulnerability -> 1 day
PublicUkraineRussiaIps -> 1 hour

How to pull all available data sources:
For pulling all available data sources you just need to define the array as follows:

Code Block
[ "ip", "hash", "domain", "url", "vulnerability", "PublicUkraineRussiaIps" ]

How to pull specific data sources:
For pulling a specific list of sources, you just need to remove them from the previous one and define those that you need. For example to pull pull hash and PublicUkraineRussiaIps sources you can define the array as follows:

Code Block
[ "hash", "PublicUkraineRussiaIps" ]

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-recorded-future-docker-image-1.2.0	`2bbfa2c4c4b1dcc6bea88d9216afcdc0d487a2ef9c01b18fa1d8a7d19b4ebcbd`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Change log for v1.x.x

Release

Released on

Release type

Details

Recommendations

v1.0.0

20 May 2021

Status

colour	Green
title	FEATURES

New features:

Initial release with the following Recorded Future Threat List default data sources:
- IPs
- Domains
- URLs
- File Hashes
- Vulnerabilities

Upgrade

v1.1.0

24 May 2022

Status

colour	Purple
title	IMPROVEMENTS

Status

colour	Yellow
title	VULNS

Improvement:

Upgrade the IFC SDK Lookup Factory Service to improve the data model validation.
The underlay Devo Collector SDK has been upgraded to v1.1.4 to improve efficiency, increase the resilience and mitigate vulnerabilities.
The hard-reset procedure when losing connection with Devo has been improved.

Vulnerabilities mitigated:

CVE-2022-1664
CVE-2021-33574
CVE-2022-23218
CVE-2022-23219
CVE-2019-8457
CVE-2022-1586
CVE-2022-1292
CVE-2022-2068
CVE-2022-1304
CVE-2022-1271
CVE-2021-3999
CVE-2021-33560
CVE-2022-29460
CVE-2022-29458
CVE-2022-0778
CVE-2022-2097
CVE-2020-16156
CVE-2018-25032

Upgrade

v1.2.0-stable

01 Jun 2022

Status

colour	Green
title	FEATURES

New features:

Added new custom data source for retrieving IPs related to Ukraine and Russia countries.

Recommended version

Configuration checklist

Here you will find a brief checklist of the important configurations that need to be done for deploying this collector:

Configuration

Requirements

Recorded Future API key

Generate your API token.

Refer to the Enable the collection section.

Version	Old Version 2	New Version 3
Changes made by	Former user	Former user
Saved on	Jun 07, 2022	Sept 08, 2022

Versions Compared

Key

Overview

Benefits

Integration architecture

Configuration

Recorded Future configuration

Devo collector features

Data sources

Vendor setup

Enable the collection

Devo configuration

Using the integration

View lookup tables

Run some sample correlation queries

Install some sample alerts

Connection to potentially malicious IPv4 Address (Firewall)

Connection to a potentially malicious domain

Minimum configuration required for basic pulling

Run the collector

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Change log for v1.x.x

Configuration checklist