Page Comparison

...

<any_directory> └── devo-collectors/ └── <product_name>/ ├── certs/

To enable the collector for a customer:

In the Collector Server GUI, access the domain in which you want this instance to be created
Click Add Collector and find the one you wish to add.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.
In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

Rw ui tabs macro

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

{
  "global_overrides": {
    "debug": <debug_status>
  },
  "inputs": {
    "recorded_future": {
     │ "id":  ├── chain.crt"<short_unique_id>",
      "enabled": <input_status>,
  │   ├── <your_domain>.key
"requests_per_second": <requests_per_second_value>,
       │"base_url": <url_value>,
  └── <your_domain>.crt   "credentials":{
     ├── state/
   "api_token": "<api_token_value>"
      },
      "services": {
        "lookup_puller": {
          "types": <list_of_sources>
       └── config/}
      }
    }
 └── config.yaml}
}

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Image Removed

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

language	yaml

globals:
  debug: <debug_status>
  id: <collector_id>
  name: <collector_name>
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  recorded_future:
    id: <input_id>
    enabled: <input_status>
    requests_per_second: <requests_per_second>
    base_url: <base_url>
    api_token: <api_token_value>
    services:
      lookup_puller:
        # Intelligence Types Not Required Can Be Removed / Commented Out
        types:
          - <list_of_sources>

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Replace the placeholders with your required values following the description table below:

Parameter

Data Type

Type

Value Range

Details

debug_status

bool

Mandatory

false / true

If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.

collector_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this collector.

collector_name

str

Mandatory

Minimum length: 1
Maximum length: 10

Use this param to give a valid name to this collector.

devo_address

str

Mandatory

collector-us.devo.io
collector-eu.devo.io

Use this param to identify the Devo Cloud where the events will be sent.

chain_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt

cert_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.cert downloaded from your Devo domain.

key_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.key downloaded from your Devo domain.

input_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

If the value is true, the input definition will be executed. If the value is false, the service will be ignored.

requests_per_second

int

Optional

Minimum value: 1

Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.

Info
This parameter should be removed if it is not used.

base_url

str (with placesholders)

Mandatory

The URL must comply with the following format:

https://api.recordedfuture.com/v2/{0}/risklist?format=csv/splunk

Use this param to define the URL used by the collector to pull data.

Take into account that {0} is a placeholder for data type.

Info
This parameter is used for `ip`, `hash`, `domain`, `url` and `vulnerability` data sources.

api_token_value

str

Mandatory

Minimum length: 1

Set up here your access token provided by Recorded Future.

list_of_sources

list of str (in yaml format)

Mandatory

Minimum data sources: 1

This configuration allows you to define what data sources will be pulled.

Info

Keep in mind that each data source has different refresh times associated with it:

ip → 1 hour
hash → 1 day
domain → 2 hours
url → 1 day
vulnerability → 1 day
PublicUkraineRussiaIps → 1 hour

How to pull all available data sources:
For pulling all available data sources you just need to define the array as follows:

Code Block
types: - ip - hash - domain - url - vulnerabilitiy - PublicUkraineRussiaIps

How to pull specific data sources:
For pulling a specific list of sources, you just need to remove them from the previous one and define those that you need. For example to pull pull hash and PublicUkraineRussiaIps sources you can define the array as follows:

Code Block
types: - hash - PublicUkraineRussiaIps

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-recorded_future_if-docker-image-1.3.0

8617831a821d0c0c2a451e3bbde41f2ac5b8c52966d1806d6cb76d83793981a4

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Lookup puller service

The only service available in this collector is lookup_puller. It contains 6 different pullers:

IpAddressLookupPuller
FileHashLookupPuller
DomainLookupPuller
UrlLookupPuller
VulnerabilityLookupPuller
PublicUkraineRussiaIpsLookupPuller

Expand

title	Devo categorization and destination

All events of this service are ingested into these tables:

IpAddressLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Address_Threat_List
- my.lookuplist.Recorded_Future_IPv6_Address_Threat_List
FileHashLookupPuller:
- my.lookuplist.Recorded_Future_Domain_Threat_List
DomainLookupPuller:
- my.lookuplist.Recorded_Future_File_Hash_Threat_List
UrlLookupPuller:
- my.lookuplist.Recorded_Future_URL_Threat_List
VulnerabilityLookupPuller:
- my.lookuplist.Recorded_Future_Vulnerability_Threat_List
PublicUkraineRussiaIpsLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Public_Ukranie_Russia_List
- my.lookuplist.Recorded_Future_IPv6_Public_Ukranie_Russia_List

Setup output

A successful run has the following output messages for the setup module:

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Code Block

INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> successfully generated new access token
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from  server
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Terminated
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Setup for module <DataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> PrePull Started. INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values. INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} WARNING InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> PrePull Terminated INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Started INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00 INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting API for issues INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from

Code Block
Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Please replace the placeholders with real world values following the description table below:

Parameter

Data Type

Type

Value Range / Format

Details

debug_status

bool

Mandatory

false / true

If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.

short_unique_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

If the value is true, the input definition will be executed. If the value is false, the service will be ignored.

requests_per_second_value

int

Optional

Minimum value: 1

Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.

This parameter should be removed if it is not used.

url_value

str

Mandatory

The URL must comply with the following format:

https://api.recordedfuture.com/v2/{0}/risklist?format=csv/splunk

Use this param to define the URL used by the collector to pull data.

Take into account that {0} is a placeholder for data type.

This parameter is used for ip, hash, domain, url and vulnerability data sources.

api_token_value

str

Mandatory

Minimum length: 1

Set up here your access token provided by Recorded Future.

list_of_sources

array

Mandatory

Minimum data sources: 1

This configuration allows you to define what data sources will be pulled.

Keep in mind that each data source has different refresh times associated with it:

ip → 1 hour
hash → 1 day
domain -> 2 hours
url -> 1 day
vulnerability -> 1 day
PublicUkraineRussiaIps -> 1 hour

How to pull all available data sources:
For pulling all available data sources you just need to define the array as follows:

Code Block
[ "ip", "hash", "domain", "url", "vulnerability", "PublicUkraineRussiaIps" ]

How to pull specific data sources:
For pulling a specific list of sources, you just need to remove them from the previous one and define those that you need. For example to pull pull hash and PublicUkraineRussiaIps sources you can define the array as follows:

Code Block
[ "hash", "PublicUkraineRussiaIps" ]

Image Added

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Image Added

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

language	yaml

globals:
  debug: <debug_status>
  id: <collector_id>
  name: <collector_name>
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  recorded_future:
    id: <input_id>
    enabled: <input_status>
    requests_per_second: <requests_per_second>
    base_url: <base_url>
    api_token: <api_token_value>
    services:
      lookup_puller:
        # Intelligence Types Not Required Can Be Removed / Commented Out
        types:
          - <list_of_sources>

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Replace the placeholders with your required values following the description table below:

Parameter

Data Type

Type

Value Range

Details

debug_status

bool

Mandatory

false / true

If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.

collector_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this collector.

collector_name

str

Mandatory

Minimum length: 1
Maximum length: 10

Use this param to give a valid name to this collector.

devo_address

str

Mandatory

collector-us.devo.io
collector-eu.devo.io

Use this param to identify the Devo Cloud where the events will be sent.

chain_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt

cert_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.cert downloaded from your Devo domain.

key_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.key downloaded from your Devo domain.

input_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

input_status

bool

Mandatory

false / true

If the value is true, the input definition will be executed. If the value is false, the service will be ignored.

requests_per_second

int

Optional

Minimum value: 1

Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.

Info
This parameter should be removed if it is not used.

base_url

str (with placesholders)

Mandatory

The URL must comply with the following format:

https://api.recordedfuture.com/v2/{0}/risklist?format=csv/splunk

Use this param to define the URL used by the collector to pull data.

Take into account that {0} is a placeholder for data type.

Info
This parameter is used for `ip`, `hash`, `domain`, `url` and `vulnerability` data sources.

api_token_value

str

Mandatory

Minimum length: 1

Set up here your access token provided by Recorded Future.

list_of_sources

list of str (in yaml format)

Mandatory

Minimum data sources: 1

This configuration allows you to define what data sources will be pulled.

Info

Keep in mind that each data source has different refresh times associated with it:

ip → 1 hour
hash → 1 day
domain → 2 hours
url → 1 day
vulnerability → 1 day
PublicUkraineRussiaIps → 1 hour

How to pull all available data sources:
For pulling all available data sources you just need to define the array as follows:

Code Block
types: - ip - hash - domain - url - vulnerabilitiy - PublicUkraineRussiaIps

How to pull specific data sources:
For pulling a specific list of sources, you just need to remove them from the previous one and define those that you need. For example to pull pull hash and PublicUkraineRussiaIps sources you can define the array as follows:

Code Block
types: - hash - PublicUkraineRussiaIps

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-recorded_future_if-docker-image-1.5.0	`59fe0f4cb3d06bce6fe570414519a24a928c08d4bcf601a21d4066452f6d411c`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Lookup puller service

The only service available in this collector is lookup_puller. It contains 6 different pullers:

IpAddressLookupPuller
FileHashLookupPuller
DomainLookupPuller
UrlLookupPuller
VulnerabilityLookupPuller
PublicUkraineRussiaIpsLookupPuller

Expand

title	Devo categorization and destination

All events of this service are ingested into these tables:

IpAddressLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Address_Threat_List
- my.lookuplist.Recorded_Future_IPv6_Address_Threat_List
FileHashLookupPuller:
- my.lookuplist.Recorded_Future_Domain_Threat_List
DomainLookupPuller:
- my.lookuplist.Recorded_Future_File_Hash_Threat_List
UrlLookupPuller:
- my.lookuplist.Recorded_Future_URL_Threat_List
VulnerabilityLookupPuller:
- my.lookuplist.Recorded_Future_Vulnerability_Threat_List
PublicUkraineRussiaIpsLookupPuller:
- my.lookuplist.Recorded_Future_IPv4_Public_Ukranie_Russia_List
- my.lookuplist.Recorded_Future_IPv6_Public_Ukranie_Russia_List

Expand

title	Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block

INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> successfully generated new access token
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from  server
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Puller Setup Terminated
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Setup for module <DataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info
Note that the `PrePull` action is executed only one time before the first run of the `Pull` action.

Code Block

INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45PrePull Started.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> RemovingUser thehas duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> DeliveringNo issuessaved tostate thefound, SDKinitializing INFOwith InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(20221970, 51, 121, 190, 13, 20, 1931910), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}.
INFOWARNING InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Saved (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20.state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> RequestingPrePull Terminated
API for issues
INFO OutputProcessInputProcess::SyslogSenderDataPuller(standard_senders,syslog_sender_0data_puller,00011,issues,predefined) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}Starting data collection every 60 seconds
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Started
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfullyFetching retriedfor issues from 2022-01-01T00:00:00
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting Removing theAPI duplicatefor issues
if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flattensuccessfully dataretried isissues setfrom to
True. Flattening the data and adding 'devo_pulling_id' to eventsINFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> DeliveringRemoving the duplicate issues toif thepresent
SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 20 issues deliveredFlatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> StateDelivering hasissues beento updatedthe duringSDK
pagination: {INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 65, 3012, 919, 013, 120, 927011193191), 'buffer_ids_with_duplication_risk': ['87e301c509992ee4-d3b71450-4c2b44fa-9495951c-9163772b3517d5fc4815473a', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}.
INFO InputProcess::]}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 21; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 4020.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting  API for issues
INFO InputProcessOutputProcess::DataPullerSyslogSender(datastandard_puller,00011,issues,senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from 
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 520 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 131, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Updating deduplication buffers content
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Terminated
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference. (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

Expand

title	Restart the persistence

This collector does not persist in any data.

...

title	Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

...

ErrorType

...

Error Id

...

Error Message

...

Cause

...

Solution

...

ConnectionError

...

-

...

Error retrieving data from API with response code {status_code}. This pull iteration did not produce any results.

...

Response’s status code is different to 200. The cause depends on the status code received. These are some of the most common status codes:

401: Unauthorized. Invalid credentials.
403: Forbidden. Not allowed to perform this action.
404: Not found. Invalid endpoint URL.
429: Too may requests. API’s Rate Limiter has been exceeded.

...

Depending on the value of the status code, solutions for the most common error could be:

401: Try valid credentials.
403: Try some credentials with privileges to make these requests.
404: Try a valid endpoint.
429: Set requests_per_second parameter in config file to a lower value.

...

LookupError

...

-

...

All lookups have been rejected. ETL aborted!

...

The collected messages has an unexpected format.

...

This errors are expected as some data sources will not match the expected and accepted format.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Expand

title	Verify collector operations

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

Code Block

INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 1000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-eu.devo.io', 'port': 443, ...}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
INFO OutputProcess::MainThread -> Process started927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Requesting  API for issues
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> successfully retried issues from 
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> 5 issues delivered
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Updating deduplication buffers content
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Pull Terminated
INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block

INFO InputProcess::DataPuller(data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.

Info
The value `@devo_pulling_id` is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that `Pull` action in Devo’s search window.

Info

Note that a Partial Statistics Report will be displayed after download a page when the pagination is required to pull all available events. Look for the report without the Partial reference. (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

Expand

title	Restart the persistence

This collector does not persist in any data.

Expand

title	Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

ErrorType

Error Id

Error Message

Cause

Solution

ConnectionError

-

Error retrieving data from API with response code {status_code}. This pull iteration did not produce any results.

Response’s status code is different to 200. The cause depends on the status code received. These are some of the most common status codes:

401: Unauthorized. Invalid credentials.
403: Forbidden. Not allowed to perform this action.
404: Not found. Invalid endpoint URL.
429: Too may requests. API’s Rate Limiter has been exceeded.

Depending on the value of the status code, solutions for the most common error could be:

401: Try valid credentials.
403: Try some credentials with privileges to make these requests.
404: Try a valid endpoint.
429: Set requests_per_second parameter in config file to a lower value.

LookupError

-

All lookups have been rejected. ETL aborted!

The collected messages has an unexpected format.

This errors are expected as some data sources will not match the expected and accepted format.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

...

Release

...

Released on

...

Release type

...

Details

...

Recommendations

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Expand

title	Verify collector operations

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

Code Block

INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 1000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-eu.devo.io', 'port': 443, ...}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
INFO OutputProcess::MainThread -> Process started
INFO InputProcess::MainThread -> Process Started
INFO InputProcess::MainThread -> InitVariables Started
INFO InputProcess::MainThread -> Validating variables in collector definitions Started
INFO InputProcess::MainThread -> Flatten data is not provided in the config.yaml. Considering the flatten data from collector definitions
INFO InputProcess::MainThread -> Validating collector Variables is terminated
INFO InputProcess::MainThread -> Initialization of api_base_url has started.
INFO InputProcess::MainThread -> api_base_url has been initialized
INFO InputProcess::MainThread -> Initialization of credentials has started.
INFO InputProcess::MainThread -> credentials have been initialized.
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO InputProcess::MainThread -> InitVariables Terminated
INFO InputProcess::MainThread -> InputThread(data_puller,111) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> ServiceThread(data_puller,111,issues,predefined) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> DataPullerSetup(collector,data_puller#111,issues#predefined) -> Starting thread
INFO InputProcess::MainThread -> Process Started
INFODataPuller(data_puller,111,issues,predefined) - Starting thread
WARNING InputProcess::MainThread -> InitVariables StartedDataPuller(data_puller,111,issues,predefined) -> Waiting until setup will be executed
INFO InputProcess::MainThreadDataPullerSetup(collector,data_puller#111,issues#predefined) -> ValidatingPuller variables in collector definitions Setup Started
INFO InputProcess::MainThreadDataPullerSetup(collector,data_puller#111,issues#predefined) -> Flatten dataThis is notthe providedfirst inrun theof configcollector.yaml. ConsideringGenerating the flattenaccess data from collector definitions
token
INFO InputProcess::MainThreadDataPullerSetup(collector,data_puller#111,issues#predefined) -> Validating collector Variables is terminatedGetting the auth token url based on provided api_base_url
INFO InputProcess::MainThreadDataPullerSetup(collector,data_puller#111,issues#predefined) -> InitializationUsing of api_base_url has started.
INFO InputProcessdefault Authentication Domain auth..io for fetching Access Token
INFO OutputProcess::MainThread -> api_base_url has been initialized
INFO InputProcess::MainThread -> Initialization of credentials has started.
INFO InputProcess::MainThread -> credentials have been initialized.[INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO InputProcessOutputProcess::MainThread -> InitVariables Terminated
INFO InputProcess::MainThread -> InputThread(data_puller,111DevoSender(standard_senders,devo_sender_0) -> Starting thread (execution_period=120s)
INFO InputProcessOutputProcess::MainThread -> ServiceThreadDevoSenderManagerMonitor(datastandard_puller,111,issues,predefinedsenders,devo_1) -> Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> DataPullerSetup(collector,data_puller#111,issues#predefined) -> Starting threadevery 300 seconds)
INFO InputProcessOutputProcess::MainThread -> DataPullerDevoSenderManager(datastandard_pullersenders,111,issues,predefinedmanager,devo_1) -> Starting thread
WARNINGINFO InputProcessOutputProcess::DataPuller(data_puller,111,issues,predefinedMainThread -> DevoSender(lookup_senders,devo_sender_0) -> Waiting until setup will be executed
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined)Starting thread
INFO OutputProcess::MainThread -> Puller Setup Started
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> This is the first run of collector. Generating the access token
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefinedDevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> GettingStarting thethread
auth token url based on provided api_base_url
INFO InputProcess::DataPullerSetup(collector,data_puller#111,issues#predefined) -> Using default Authentication Domain auth..io for fetching Access TokenINFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__DevoSenderManager(internal_senders,manager,devo_1) -> Starting Thethread
<address> does not appear to be an IP address and cannot be verified: collector-eu.devo.ioINFO InputProcess::MainThread -> [GC] global: 36.7% -> 36.7%, process: RSS(26.93MiB -> 27.97MiB), VMS(334.43MiB -> 334.67MiB)
INFO OutputProcess::MainThread -> [INTERNAL LOGICGC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_senderglobal: 36.7% -> 36.3%, process: RSS(26.68MiB -> 28.61MiB), VMS(910.71MiB -> 910.71MiB)
INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-eu.devo.io:443", ...}

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Code Block

INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> StartingNumber threadof INFOavailable OutputProcesssenders::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread 1, sender manager internal queue size: 0
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookupSyslogSenderManagerMonitor(standard_senders,devosidecar_10) -> Starting thread (every 300 seconds)
enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::MainThread -> DevoSenderManager(lookupSyslogSenderManagerMonitor(standard_senders,manager,devosidecar_10) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(internalSender: SyslogSender(standard_senders,devosyslog_sender_0) -> Starting thread, status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internalSyslogSenderManagerMonitor(standard_senders,devosidecar_10) -> StartingStandard thread- (everyTotal 300number seconds)of INFOmessages OutputProcesssent::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
INFO InputProcess::MainThread -> [GC] global: 36.7% -> 36.7%, process: RSS(26.93MiB -> 27.97MiB), VMS(334.43MiB -> 334.67MiB 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::MainThreadSyslogSenderManagerMonitor(internal_senders,sidecar_0) -> [GC] global: 36.7% -> 36.3%, process: RSS(26.68MiB -> 28.61MiB), VMS(910.71MiB -> 910.71MiB) Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::DevoSenderSyslogSenderManagerMonitor(internal_senders,devosidecar_sender_0) -> Created a senderSender: SyslogSender(internal_senders,syslog_sender_0), status: {"groupinternal_queue_namesize": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders", "instance_name": "devo_sender_0", "url": "collector-eu.devo.io:443", ...}

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Code Block

INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)

Info
By default, these information traces will be displayed every 10 minutes.

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services	Description
`internal_senders`	In charge of delivering internal metrics to Devo such as logging traces or metrics.
`standard_senders`	In charge of delivering pulled events to Devo.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

Info
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00":

44 (elapsed 0.007 seconds) INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0 INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {} INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)

Info
By default, these information traces will be displayed every 10 minutes.

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services

Description

internal_senders

In charge of delivering internal metrics to Devo such as logging traces or metrics.

standard_senders

In charge of delivering pulled events to Devo.

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

Info
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.

Info
By default these traces will be shown every 10 minutes.

Expand

title	Check memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

Code Block

INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)

Info

Differences between RSS and VMS memory usage:

RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using

Expand

title	Enable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Change log

21 (elapsed 0.007 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.

Info
By default these traces will be shown every 10 minutes.

Expand

title	Check memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

Code Block

INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)

Info

Differences between RSS and VMS memory usage:

RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using

Expand

title	Enable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Change log

Release

Released on

Release type

Details

Recommendations

v1.5.0

12 Dec 2023

Status

colour	Green
title	IMPRoVEMENTs

Updated DCSDK from 1.10.2 to 1.12.4:
- Fixed error related a ValueError exception not well controlled.
- Fixed error related with loss of some values in internal messages(collector_name, collector_id and job_id)
- Improve Controlled stop when InputProcess is killed
- Change internal queue management for protecting against OOMK
- Extracted ModuleThread structure from PullerAbstract
- Improve Controlled stop when both processes fails to instatiate
- Upgrade DevoSDK dependency to version v5.4.0
- Fixed error in persistence system
- Applied changes to make DCSDK compatible with MacOS
- Added new sender for relay in house + TLS
- Added persistence functionality for gzip sending buffer
- Added Automatic activation of gzip sending
- Improved behaviour when persistence fails
- Upgraded DevoSDK dependency
- Fixed console log encoding
- Restructured python classes
- Improved behaviour with non-utf8 characters
- Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)
- New persistence format/structure (compression in some cases)
- Removed dmesg execution (It was invalid for docker execution)
- Added extra check for not valid message timestamps
- Added extra check for improve the controlled stop
- Changed default number for connection retries (now 7)
- Fix for Devo connection retries
- Updated DevoSDK to v5.1.10
- Fix for SyslogSender related to UTF-8
- Enhace of troubleshooting. Trace Standardization, Some traces has been introduced.
- Introduced a machanism to detect "Out of Memory killer" situation.
- Updated DevoSDK to v5.1.9
- Fixed some bug related to development on MacOS
- Added an extra validation and fix when the DCSDK receives a wrong timestamp format
- Added an optional config property for use the Syslog timestamp format in a strict way
Updated the docker image version to 1.3.0.

Recommended version

v1.4.0

12 Dec 2023

Status

colour	Green
title	IMPRoVEMENTs

Status

colour	Red
title	BUG FIXING

Fixed a credentials config bug.
Updated DCSDK from 1.9.1 to 1.10.2 . Recorded Future v1.4.0 Release Notes

Upgrade

v1.3.0

16 Aug 2023

Status

colour	Green
title	IMPRoVEMENTIMPRoVEMENTs

Updated DCSDK from 1.1.4 to 1.9.1: https://devoinc.atlassian.net/wiki/spaces/IF/pages/3901620225

Recommended version

Upgrade

v1.2.0-stable

01 Jun 2022

Status

colour	GreenBlue
title	new FEATURES

Added new custom data sources from Recorded Future Threat List service:

IPs related to Ukraine and Russia countries.

Recommended version

Upgrade

v1.1.0

24 May 2022

Status

colour	GreenBlue
title	new FEATURES

Status

colour	Yellow
title	VULNERABILITIES

This release includes:

Upgrade the base docker-image from Debian to Ubuntu20 for vulnerabilities mitigation.
Upgrade the IFC SDK Lookup Factory Service to improve the data model validation.

Upgrade to v1.2.0

v1.0.2

20 May 2021

Status

colour	GreenBlue
title	new FEATURES

Initial release with the following Recorded Future Threat List default data sources:

IPs
Domains
URLs
File Hashes
Vulnerabilities

Upgrade to v1.2.0

Versions Compared

Old Version 23

New Version Current

Key

Editing the JSON configuration

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Collector services detail

Lookup puller service

Setup output

Puller output

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Collector services detail

Lookup puller service

Setup output

Puller output

Collector operations

Initialization

Collector operations

Sender statistics

Initialization

Events delivery and Devo ingestion

Events delivery and Devo ingestion

Sender services

Sender statistics

Sender services

Change log

Change log