Recorded Future is a threat intelligence provider that allows you to access known bad incidents of compromise and entity enrichment capabilities. It has 6 different modules and charges on a per-user basis for access to the product. The 6 different modules are:
Security Operations: Providing intel into SIEM / SOAR platforms.
Standalone Threat Intelligence: An extension of Security Operations providing context and enrichment of known and emerging threats/incidents.
Brand Intelligence: Monitoring an organization’s external exposure.
Vulnerability Management: Intelligence into the prioritization of threats.
Third party intelligence: Data from third party sources.
Geo-political: More focussed on nation-state attacks and threat indicators.
Note
Recorded Future also charges customers for each integration they use. For example, a mutual customer of Recorded Future and Devo using this integration will pay Recorded Future a subscription fee.
Devo collector features
...
Feature
...
Details
...
Allow parallel downloading (multipod)
...
Not allowed
...
Running environments
...
Collector server
On-premise
...
Populated Devo events
...
Lookups
Data sources
...
Data Source
...
Description
...
API Endpoint
...
Collector service name
...
Devo Table
...
Available from release
...
IpAddressLookupPuller
...
This endpoint provides a list of IPs classified as Threat by Recorded Future.
This collector only requires a source configuration by generating a Recorded Future API key. Recorded Future clients with Advanced or Core licenses, and one or more Connect API subscriptions can create and manage their API tokens directly in the Recorded Future portal.
Enable the collection
Here you find the necessary steps to enable the collection.
Steps
...
Screenshots
...
1
...
Login to your Recorded Future account. Open the Menu in the upper-right corner and select the option User Settings.
...
2
...
Select the API Access tab. To create a new API token, click on Generate New API Token.
...
3
...
Enter a name for the token in the Name field. Then click on Generate.
...
-
...
4
...
Select Devo from the drop-down integration list
...
-
...
5
...
Click on the Generate new API token button. The new API token appears in the table below. Make a note of the token value as this is required for the ingest Configuration.
-
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
Info
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
...
Setting
...
Details
...
url_value
...
This param refers to the endpoint used by the collector to pull data.
...
api_token_value
...
This is the access token provided by Recorded Future.
...
list_of_sources
...
This configuration allows you to define what data sources will be pulled.
Accepted authentication methods
...
Authentication Method
...
Token
...
Token
...
Status
colour
Green
title
REQUIRED
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
...
Rw tab
title
Cloud collector
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.
Rw tab
title
On-premise collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.
Replace the placeholders with your required values following the description table below:
...
Parameter
...
Data Type
...
Type
...
Value Range
...
Details
...
debug_status
...
bool
...
Mandatory
...
false / true
...
If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.
...
collector_id
...
int
...
Mandatory
...
Minimum length: 1 Maximum length: 5
...
Use this param to give an unique ID to this collector.
...
collector_name
...
str
...
Mandatory
...
Minimum length: 1 Maximum length: 10
...
Use this param to give a valid name to this collector.
...
devo_address
...
str
...
Mandatory
...
collector-us.devo.io collector-eu.devo.io
...
Use this param to identify the Devo Cloud where the events will be sent.
...
chain_filename
...
str
...
Mandatory
...
Minimum length: 4 Maximum length: 20
...
Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt
...
cert_filename
...
str
...
Mandatory
...
Minimum length: 4 Maximum length: 20
...
Use this param to identify the file.cert downloaded from your Devo domain.
...
key_filename
...
str
...
Mandatory
...
Minimum length: 4 Maximum length: 20
...
Use this param to identify the file.key downloaded from your Devo domain.
...
input_id
...
int
...
Mandatory
...
Minimum length: 1 Maximum length: 5
...
Use this param to give an unique id to this input service.
Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
...
input_status
...
bool
...
Mandatory
...
false / true
...
If the value is true, the input definition will be executed. If the value is false, the service will be ignored.
...
requests_per_second
...
int
...
Optional
...
Minimum value: 1
...
Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.
Info
This parameter should be removed if it is not used.
How to pull specific data sources: For pulling a specific list of sources, you just need to remove them from the previous one and define those that you need. For example to pull pull hash and PublicUkraineRussiaIps sources you can define the array as follows:
Code Block
[
"hash",
"PublicUkraineRussiaIps"
]
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Replace <product_name>, <image_name> and <version> with the proper values.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:
Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note
Replace <product_name>, <image_name> and <version> with the proper values.
Change log for v1.x.x
...
Release
...
Released on
...
Release type
...
Details
...
Recommendations
...
v1.0.0
...
...
Status
colour
Green
title
FEATURES
...
New features:
Initial release with the following Recorded Future Threat List default data sources:
IPs
Domains
URLs
File Hashes
Vulnerabilities
...
Upgrade
...
v1.1.0
...
Status
colour
Purple
title
IMPROVEMENTS
Status
colour
Yellow
title
VULNS
...
Improvement:
Upgrade the IFC SDK Lookup Factory Service to improve the data model validation.
The underlay Devo Collector SDK has been upgraded to v1.1.4 to improve efficiency, increase the resilience and mitigate vulnerabilities.
The hard-reset procedure when losing connection with Devo has been improved.
Vulnerabilities mitigated:
CVE-2022-1664
CVE-2021-33574
CVE-2022-23218
CVE-2022-23219
CVE-2019-8457
CVE-2022-1586
CVE-2022-1292
CVE-2022-2068
CVE-2022-1304
CVE-2022-1271
CVE-2021-3999
CVE-2021-33560
CVE-2022-29460
CVE-2022-29458
CVE-2022-0778
CVE-2022-2097
CVE-2020-16156
CVE-2018-25032
...
Upgrade
...
v1.2.0-stable
...
...
Status
colour
Green
title
FEATURES
...
New features:
Added new custom data source for retrieving IPs related to Ukraine and Russia countries.
...
Recommended version
Configuration checklist
Here you will find a brief checklist of the important configurations that need to be done for deploying this collector:
...
Configuration
...
Requirements
...
Recorded Future API key
Generate your API token.
...
Table of Contents
maxLevel
2
minLevel
2
type
flat
Configuration requirements
To run this collector, there are some configurations detailed below that you need to take into account.
Configuration
Details
Recorded Future API key
Generate your API token.
Info
Refer to the Vendor setup section to know more about these configurations.
Overview
Recorded Future is a threat intelligence provider that allows you to access known bad incidents of compromise and entity enrichment capabilities. It has 6 different modules and charges on a per-user basis for access to the product. The 6 different modules are:
Security Operations: Providing intel into SIEM / SOAR platforms.
Standalone Threat Intelligence: An extension of Security Operations providing context and enrichment of known and emerging threats/incidents.
Brand Intelligence: Monitoring an organization’s external exposure.
Vulnerability Management: Intelligence into the prioritization of threats.
Third party intelligence: Data from third-party sources.
Geo-political: More focussed on nation-state attacks and threat indicators.
Note
Recorded Future also charges customers for each integration they use. For example, a mutual customer of Recorded Future and Devo using this integration will pay Recorded Future a subscription fee.
Devo collector features
Feature
Details
Allow parallel downloading (multipod)
Not allowed
Running environments
Collector server
On-premise
Populated Devo events
Lookups
Flattening Preprocessing
No
Data sources
Data Source
Description
API Endpoint
Collector service name
Devo Table
Available from release
IpAddressLookupPuller
This endpoint provides a list of IPs classified as Threat by Recorded Future.
Click on Menu and select the option User Settings.
Select the API Access tab.
To create a new API token, click on Generate New API Token.
Enter a name for the token.
Select Devo from the integration list.
Click on the Generate new API token button.
Make a note of the token value, as this is required for the Ingest Configuration.
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
Info
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting
Details
url_value
This param refers to the endpoint used by the collector to pull data.
api_token_value
This is the access token provided by Recorded Future.
list_of_sources
This configuration allows you to define what data sources will be pulled.
Info
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.
Accepted authentication methods
Authentication Method
Token
Token
Status
colour
Green
title
REQUIRED
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Rw ui tabs macro
Rw tab
title
Cloud collector
We use a piece of software called Collector Server to host and manage all our available collectors.
To enable the collector for a customer:
In the Collector ServerGUI, access the domain in which you want this instance to be created
Click Add Collector and find the one you wish to add.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.
In the Parameters section, establish the Collector Parameters as follows below:
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.
Please replace the placeholders with real world values following the description table below:
Parameter
Data Type
Type
Value Range / Format
Details
debug_status
bool
Mandatory
false / true
If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.
short_unique_id
int
Mandatory
Minimum length: 1 Maximum length: 5
Use this param to give an unique id to this input service.
Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
input_status
bool
Mandatory
false / true
If the value is true, the input definition will be executed. If the value is false, the service will be ignored.
requests_per_second_value
int
Optional
Minimum value: 1
Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.
This parameter should be removed if it is not used.
How to pull specific data sources: For pulling a specific list of sources, you just need to remove them from the previous one and define those that you need. For example to pull pull hash and PublicUkraineRussiaIps sources you can define the array as follows:
Code Block
[
"hash",
"PublicUkraineRussiaIps"
]
Image Added
Rw tab
title
On-premise collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.
Replace the placeholders with your required values following the description table below:
Parameter
Data Type
Type
Value Range
Details
debug_status
bool
Mandatory
false / true
If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.
collector_id
int
Mandatory
Minimum length: 1 Maximum length: 5
Use this param to give an unique id to this collector.
collector_name
str
Mandatory
Minimum length: 1 Maximum length: 10
Use this param to give a valid name to this collector.
devo_address
str
Mandatory
collector-us.devo.io collector-eu.devo.io
Use this param to identify the Devo Cloud where the events will be sent.
chain_filename
str
Mandatory
Minimum length: 4 Maximum length: 20
Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt
cert_filename
str
Mandatory
Minimum length: 4 Maximum length: 20
Use this param to identify the file.cert downloaded from your Devo domain.
key_filename
str
Mandatory
Minimum length: 4 Maximum length: 20
Use this param to identify the file.key downloaded from your Devo domain.
input_id
int
Mandatory
Minimum length: 1 Maximum length: 5
Use this param to give an unique id to this input service.
Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
input_status
bool
Mandatory
false / true
If the value is true, the input definition will be executed. If the value is false, the service will be ignored.
requests_per_second
int
Optional
Minimum value: 1
Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.
Info
This parameter should be removed if it is not used.
How to pull specific data sources: For pulling a specific list of sources, you just need to remove them from the previous one and define those that you need. For example to pull pull hash and PublicUkraineRussiaIps sources you can define the array as follows:
Code Block
types:
- hash
- PublicUkraineRussiaIps
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Replace <product_name>, <image_name> and <version> with the proper values.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> successfully generated new access token
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from server
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Terminated
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Setup for module <DataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> PrePull Started.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
WARNING InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> PrePull Terminated
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Started
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting API for issues
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting API for issues
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting API for issues
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 5 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Updating deduplication buffers content
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Terminated
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Info
Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference. (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
Expand
title
Restart the persistence
This collector does not persist in any data.
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
ErrorType
Error Id
Error Message
Cause
Solution
ConnectionError
-
Error retrieving data from API with response code {status_code}. This pull iteration did not produce any results.
Response’s status code is different to 200. The cause depends on the status code received. These are some of the most common status codes:
401: Unauthorized. Invalid credentials.
403: Forbidden. Not allowed to perform this action.
404: Not found. Invalid endpoint URL.
429: Too may requests. API’s Rate Limiter has been exceeded.
Depending on the value of the status code, solutions for the most common error could be:
401: Try valid credentials.
403: Try some credentials with privileges to make these requests.
404: Try a valid endpoint.
429: Set requests_per_second parameter in config file to a lower value.
LookupError
-
All lookups have been rejected. ETL aborted!
The collected messages has an unexpected format.
This errors are expected as some data sources will not match the expected and accepted format.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Expand
title
Verify collector operations
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
Code Block
INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 1000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-eu.devo.io', 'port': 443, ...}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
INFO OutputProcess::MainThread -> Process started
INFO InputProcess::MainThread -> Process Started
INFO InputProcess::MainThread -> InitVariables Started
INFO InputProcess::MainThread -> Validating variables in collector definitions Started
INFO InputProcess::MainThread -> Flatten data is not provided in the config.yaml. Considering the flatten data from collector definitions
INFO InputProcess::MainThread -> Validating collector Variables is terminated
INFO InputProcess::MainThread -> Initialization of api_base_url has started.
INFO InputProcess::MainThread -> api_base_url has been initialized
INFO InputProcess::MainThread -> Initialization of credentials has started.
INFO InputProcess::MainThread -> credentials have been initialized.
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO InputProcess::MainThread -> InitVariables Terminated
INFO InputProcess::MainThread -> InputThread(data_puller,111) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> ServiceThread(data_puller,111,issues,predefined) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> DataPullerSetup(collector,data_puller#111,issues#predefined) -> Starting thread
INFO InputProcess::MainThread -> DataPuller(data_puller,111,issues,predefined) - Starting thread
WARNING InputProcess::DataPuller(data_puller,111,issues,predefined) -> Waiting until setup will be executed
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> This is the first run of collector. Generating the access token
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Getting the auth token url based on provided api_base_url
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Using default Authentication Domain auth..io for fetching Access Token
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
INFO InputProcess::MainThread -> [GC] global: 36.7% -> 36.7%, process: RSS(26.93MiB -> 27.97MiB), VMS(334.43MiB -> 334.67MiB)
INFO OutputProcess::MainThread -> [GC] global: 36.7% -> 36.3%, process: RSS(26.68MiB -> 28.61MiB), VMS(910.71MiB -> 910.71MiB)
INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-eu.devo.io:443", ...}
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
Code Block
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)
Info
By default, these information traces will be displayed every 10 minutes.
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:
Sender services
Description
internal_senders
In charge of delivering internal metrics to Devo such as logging traces or metrics.
standard_senders
In charge of delivering pulled events to Devo.
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace
Description
Number of available senders: 1
Displays the number of concurrent senders available for the given Sender Service.
sender manager internal queue size: 0
Displays the items available in the internal sender queue.
Info
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.
Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)
Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.
Info
By default these traces will be shown every 10 minutes.
Expand
title
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using
Expand
title
Enable/disable the logging debug mode
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.
Change log
Release
Released on
Release type
Details
Recommendations
v1.5.0
Status
colour
Green
title
IMPRoVEMENTs
Updated DCSDK from 1.10.2 to 1.12.4:
Fixed error related a ValueError exception not well controlled.
Fixed error related with loss of some values in internal messages(collector_name, collector_id and job_id)
Improve Controlled stop when InputProcess is killed
Change internal queue management for protecting against OOMK
Extracted ModuleThread structure from PullerAbstract
Improve Controlled stop when both processes fails to instatiate
Upgrade DevoSDK dependency to version v5.4.0
Fixed error in persistence system
Applied changes to make DCSDK compatible with MacOS
Added new sender for relay in house + TLS
Added persistence functionality for gzip sending buffer
Added Automatic activation of gzip sending
Improved behaviour when persistence fails
Upgraded DevoSDK dependency
Fixed console log encoding
Restructured python classes
Improved behaviour with non-utf8 characters
Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)
New persistence format/structure (compression in some cases)
Removed dmesg execution (It was invalid for docker execution)
Added extra check for not valid message timestamps
Added extra check for improve the controlled stop
Changed default number for connection retries (now 7)
Fix for Devo connection retries
Updated DevoSDK to v5.1.10
Fix for SyslogSender related to UTF-8
Enhace of troubleshooting. Trace Standardization, Some traces has been introduced.
Introduced a machanism to detect "Out of Memory killer" situation.
Updated DevoSDK to v5.1.9
Fixed some bug related to development on MacOS
Added an extra validation and fix when the DCSDK receives a wrong timestamp format
Added an optional config property for use the Syslog timestamp format in a strict way
