| Table of Contents |
|---|
| minLevel | 2 |
|---|
| maxLevel | 2 |
|---|
| type | flat |
|---|
|
...
To run this collector, there are some configurations detailed below that you need to consider.
Configuration | Details |
|---|
Cylance APP | You need to run a Cylance app. |
Application ID | Once you create the App, it gives you an Application ID. |
Application Secret | Once you create the App, it gives you an Application Secret. |
Tenant ID | You can find it in your Cylance console. |
| |
| |
| |
| |
| Info |
|---|
| Info |
|---|
More information Refer to the Vendor setup section to know more about these configurations. |
Overview
SOCRadar is a cloud-based, AI-powered Digital Risk Protection Platform enhanced by cyber threat intelligence capabilities.
Devo collector features
Feature | Details |
|---|
Allow parallel downloading (multipod) | not allowedAllowed
allowed
|
Running environments | collector Collector server
onOn-premise
|
Populated Devo events | tableTable
lookupsLookups
|
Flattening preprocessing | yesNo
no
|
Data sources
Source Endpointendpoint | Collector service name | Devo |
|---|
Tabletable | Available from release |
|---|
Access History | This topic provides concepts on the user access history in Snowflake. | SELECT * FROM access_history
| access_history
| db.snowflake.history.access
| v1.0.0
|
Login History | This topic is used to query login attempts by Snowflake users within the last 365 days (1 year). | SELECT * FROM login_history
| login_history
| db.snowflake.history.login
| v1.0.0
|
Sessions | This topic provides information on the session, including information on the authentication method to Snowflake and the Snowflake login event. | SELECT * FROM sessions
| sessions
| db.snowflake.history.session
| v1.0.0
|
Custom SQL | This topic provides the chance to perform a custom SQL query | {custom_query}
| custom_service
| my.app.{custom_level_1}.{custom_level_2}
| v1.0.0
|
Data source | Description | API endpoint | Collector service name | Devo table | Available from release |
|---|
Pulled source | Description and links if needed | Used API endpoint
| Collector service to use to pull this data
| level1.level2.level3.level4
| v0.0
|
For more information on how the events are parsed, visit our page ← LINK TO THE PARSER ARTICLE IF EXISTS
Flattening preprocessing
...
Data source
...
Collector service
...
Optional
...
Flattening details
...
Source
...
Service
...
...
Flattening steps
Vendor setup
-
Minimum configuration required for basic pulling
...
Incidents | SOCRadar incidents | /company/{company_id}/incidents/v2
| incidents | threatintel.socradar.xti.incidents.1.json
| v1.0 |
Audit logs | SOCRadar audit events | /company/{company_id}/auditlogs
| audit_logs | threatintel.socradar.xti.audit_logs.1.json
| v1.0 |
Threat feeds | SOCRadar threat feed entries | /threat/intelligence/socradar_collections
| threat_feed | Lookups: socradar_threat_intelligence_urls
socradar_threat_intelligence_hashes
socradar_threat_intelligence_domains
socradar_threat_intelligence_ips
| v1.0 |
Flattening preprocessing
Data source | Collector service | Optional | Flattening details |
|---|
Incidents | incidents | | Not applicable |
Audit logs | audit_logs | | |
Threat feeds | threat_feed | | |
Vendor setup
There are some requirements to set up this collector, you will need to have an API key and create a Token. Follow the steps below:
Log into the SOCRadar portal.
Navigate to the API Options tab.
Generate a new API Key using the Actions button under the API keys section.
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
| Info |
|---|
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details. |
-
| -api_key | The API key from SOCRADAR. |
| Info |
|---|
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method. |
Accepted authentication methods
-
Run the collector
...
Authentication method | api_key | company_id |
|---|
api_key/company_id | | |
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
...
Error type
...
Error ID
...
Error message
...
Cause
...
Solution
...
SetupError
...
1
...
api_base_url must be specified
...
The api_base_url setting is missing.
...
Make sure the api_base_url setting is present under the events service in your configuration.
...
2
...
api_base_url not of expected type: str
...
The api_base_url setting has a type other than string.
...
Make sure the api_base_url setting is a string.
...
3
...
api_base_url must match one of two regexes: [...]
...
The api_base_url setting does not follow the expected format.
...
Make sure your api_base_url has this format: http[s]://{ip_address_or_domain}{:optional_port}
...
4
...
Required setting, credentials not found in user configuration
...
There is no credentials section in your input settings.
...
Make sure there is a credentials section under the threatquotient_data_puller input in your configuration.
...
5
...
Required setting, credentials not of expected type: dict
...
The credentials section is empty or has a simple type (is not an object).
...
Make sure the credentials section has a username and password fields.
...
6
...
Required setting, username not found in user configuration
...
The username setting is missing.
...
Make sure the username setting from the credentials section has a value.
...
7
...
Required setting, username not of expected type: str
...
The username setting has a type other than string.
...
Make sure the username setting from the credentials section is a string.
...
8
...
Required setting, password not found in user configuration
...
The password setting is missing.
...
Make sure the password setting from the credentials section has a value.
...
9
...
Required setting, password not of expected type: str
...
The password setting has a type other than string.
...
Make sure the password setting from the credentials section is a string.
...
10
...
Optional setting, verify_host_ssl_cert not of expected type: bool
...
The verify_host_ssl_cert setting has a type other than boolean.
...
Make sure the verify_host_ssl_cert setting is a boolean value (true/false).
...
11
...
event_fetch_limit_in_items must be greater than or equal to [...] and less than equal to [...]
...
The event_fetch_limit_in_items setting has a value too low or too high for the specified limits.
...
Make sure the event_fetch_limit_in_items setting is an integer ranged between the specified limits.
...
12
...
devo_tag_map must have an entry named "default"
...
This error is not expected to happen in a regular flow.
...
This needs to be troubleshooted by the colllector’s developers.
...
13
...
Required setting, reset_persistence_auth not of expected type: str
...
The reset_persistence_auth setting has a value, but its type is other than string.
...
Make sure the reset_persistence_auth setting is a string.
...
14
...
Required setting, historical_poll_datetime not of expected type: str
...
The historical_poll_datetime setting has a type other than string.
...
Make sure the historical_poll_datetime setting is a string.
...
15
...
historical_poll_datetime does not match expected format [...]
...
The historical_poll_datetime setting does not look like a valid date.
...
Make sure the historical_poll_datetime setting meets the mentioned format (a reference of this representation can be found here).
...
16
...
Please enter valid date for historical_poll_datetime less than or equal to one year
...
The historical_poll_datetime setting is a date older than one year.
...
Make sure the historical_poll_datetime setting does not represent a date older than one year.
...
17
...
Please enter valid date for historical_poll_datetime less than or equal to the current date
...
The historical_poll_datetime setting is a future date.
...
Make sure the historical_poll_datetime setting does not represent a future date.
...
InitVariablesError
...
100
...
Unexpected status code when fetching ThreatQuotient JWT: [...]
...
When a token was retrieved, the response had an unexpected error code.
...
Make sure your credentials are correct.
...
101
...
Unexpected status code when fetching ThreatQuotient client_id: [...]
...
The collector is having issues connecting to the ThreatQ instance.
...
Make sure you have properly configured the api_base_url setting and that you can access the {api_base_url}/assets/js/config.js URL.
...
102
...
Cannot parse client_id from ThreatQuotient server
...
The collector was expecting to find the Client’s ID, but could not find it. This is likely because the ThreatQ has been upgraded and the collector does not support it.
...
This needs to be troubleshooted by the colllector’s developers.
...
ApiError
...
400
...
Unexpected status code when fetching ThreatQuotient events: [...]
...
This error happens when the collector tries to fetch the ThreatQ events from its REST API.
...
In this error you will find the HTTP error code as long as the response’s text. This information should be enough to understand why is the error happening. Otherwise, please contact support.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
...
| title | Verify collector operations |
|---|
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
| Code Block |
|---|
INFO MainThread -> (CollectorMultithreadingQueue) standard_queue_multithreading -> max_size_in_messages: 10000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
WARNING MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
WARNING MainThread -> [OUTPUT] OutputLookupSenders -> <threshold_for_using_gzip_in_transport_layer> setting has been modified from 1.1 to 1.0 due to this configuration increases the Lookup sender performance.
WARNING MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
INFO MainThread -> [OUTPUT] OutputMultithreadingController(threatquotient_collector) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 600 seconds)
INFO MainThread -> [OUTPUT] DevoSenderManager(standard_senders,manager,devo_1)(devo_1) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 600 seconds)
INFO MainThread -> [OUTPUT] DevoSenderManager(lookup_senders,manager,devo_1)(devo_1) -> Starting thread
INFO MainThread -> InitVariables Started
INFO MainThread -> start_time_value initialized
INFO MainThread -> verify_host_ssl_cert initialized
INFO MainThread -> event_fetch_limit_in_items initialized
INFO MainThread -> InitVariables Terminated
INFO MainThread -> [INPUT] InputMultithreadingController(threatquotient_collector) - Starting thread (executing_period=300s)
INFO MainThread -> [INPUT] InputThread(threatquotient_collector,threatquotient_data_puller#111) - Starting thread (execution_period=600s)
INFO MainThread -> [INPUT] ServiceThread(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread (execution_period=600s)
INFO MainThread -> [SETUP] ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
INFO MainThread -> [INPUT] ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
|
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
| Code Block |
|---|
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)
|
| Info |
|---|
By default, these information traces will be displayed every 10 minutes. |
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:
...
Sender services
...
Description
...
internal_senders
...
In charge of delivering internal metrics to Devo such as logging traces or metrics.
...
standard_senders
...
In charge of delivering pulled events to Devo.
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
...
Logging trace
...
Description
...
Number of available senders: 1
...
Displays the number of concurrent senders available for the given Sender Service.
...
sender manager internal queue size: 0
...
Displays the items available in the internal sender queue.
| Info |
|---|
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
...
Standard - Total number of messages sent: 57, messages sent since "2023-01-10 16:09:16.116750+00:00": 0 (elapsed 0.000 seconds
...
Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2023-01-10 16:09:16.116750+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.00 seconds to be delivered.
| Info |
|---|
By default these traces will be shown every 10 minutes. |
| Expand |
|---|
|
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process. The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector. The global pressure of the available memory is displayed in the global value. All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
| Code Block |
|---|
INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)
|
| Info |
|---|
Differences between RSS and VMS memory usage: RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using
|
|
| Expand |
|---|
| title | Enable/disable the logging debug mode |
|---|
|
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes. To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector. To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode. |
Change log for v1.x.x
...
Release
...
Released on
...
Release type
...
Details
...
Recommendations
...
v1.0.0
...
Jan 1, 1990
...
NEW FEATURE BUG FIXVULNIMPROVEMENT
...
New features:
Improvements:
Bug Fixes:
The “Panic” protocol has been fixed and now works well.
Vulnerabilities Mitigation:
Critical - AlienTrojan 1345
High - AlienTrojan 154
Medium - AlienTrojan 0024
Low - AlienTrojan 5882
...
| Rw ui tabs macro |
|---|
| Rw tab |
|---|
| title | On-premise collector |
|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector: | Code Block |
|---|
<any_directory>
└── devo-collectors/
└── <product_name>/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── state/
└── config/
└── config.yaml |
| Note |
|---|
Replace <product_name> with the proper value. |
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.  | Note |
|---|
Replace <product_name> with the proper value. |
Editing the config.yaml file| Code Block |
|---|
globals:
debug: false<debug_status>
id: not<collector_usedid>
name: threatquotient<collector_collectorname>
persistence:
type: filesystem
config:
directory_name: state
outputs:
devo_us_1:
type: devo_platform
config:
address: <your-ingestion-endpoint><devo_address>
port: 443
type: SSL
chain: chain.crt<chain_filename>
cert: <your<cert_domain>.crtfilename>
key: <your<key_domain>.keyfilename>
inputs:
threatquotient_data_pullersocradar:
id: <short_unique<input_id>
debugenabled: <enable_debug_logs>true
enabledcredentials:
<input_status> requests_per_second: <requests_per_second>api_key: <api_key>
company_id: <company_id>
credentialsrequests_limits:
username- period: <threatq_username>1m
passwordnumber_of_requests: <threatq<request_per_password>minute>
services:
eventsincidents:
request_periodtag: <tag>
initial_start_time_in_secondsutc: <request<initial_start_periodtime_in_seconds>utc>
reset_persistence_auth: <reset_persistence_auth>audit_logs:
tag: <tag>
api_base_url: <api_base_url> verify_host_ssl_cert: <verify_host_ssl_cert>initial_start_time_in_utc: <initial_start_time_in_utc>
historical_poll_datetimethreat_feed:
<historical_poll_datetime> eventinitial_fetchstart_limittime_in_itemsutc: <event<initial_fetchstart_limit_in_items> |
| Info |
|---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Replace the placeholders with your required values following the description table below: |
Parameter | Data type | Type | Value range / Format | Details |
|---|
<short_unique_id>
| int
| Mandatory
| Minimum length: 1
Maximum length: 5 | Use this param to give a unique id to this input service. | Note |
|---|
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. |
|
<enable_debug_logs>
| bool
| Mandatory
| false / true
| This will make the collector generate (or not) log messages with the DEBUG level. |
<input_status>
| bool
| Mandatory
| false / true
| Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored. |
<requests_per_second>
| int
| Optional
| Minimum value: 1 | Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec. | Info |
|---|
This parameter should be removed if it is not used. |
|
<threatq_username>
| str
| Mandatory
| Any
| Username to authenticate the service. It must belong to an existing user or the initial one created during the setup. |
<threatq_password>
| str
| Mandatory
| Any
| Password to authenticate the service. It must belong to an existing user or the initial one created during the setup. |
<request_period_in_seconds>
| int
| Optional
| Minimum value: 1 | The amount (in seconds) in which the service’s collection is scheduled. | Info |
|---|
This parameter should be removed if it is not used. |
|
<reset_persistence_auth>
| str
| Optional
| Any. Recommended: date format YYYY-MM-DD | This parameter allows you to clear the persistence of the collector and restart the download pipeline. Updating this value will produce the loss of all persisted data and current pipelines. | Info |
|---|
This parameter should be removed if it is not used. |
|
<api_base_url>
| str
| Mandatory
| Must be a valid URL and comply one of these Regular Expressions: FQDN (domain): ((http|https):\/\/)([\da-z\.-]+)\.([a-z\.])([\/\w \.-]*)*([a-z])(:\d{1,5})?$ IP address: ((http|https):\/\/)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(:\d{1,5})?$
| This parameter defines the url where the ThreatQ API is available. | Info |
|---|
It has the form of http[s]://{ip_address_or_domain}{:optional_port}. |
| Note |
|---|
Note that this address must be reachable by the collector instance. |
|
<verify_host_ssl_cert>
| bool
| Mandatory
| false / true
| This should be enabled if the ThreatQ’s instance has a self-signed certificate. The usual installation steps do not include certificate signing, so this usually should be false. |
<historical_poll_datetime>
| str
| Optional
| A date with format YYYY-MM-DD HH:mm:ss. The date must be between the current date and one year ago. | This parameter allows you to clear the persistence of the collector and restart the download pipeline. | Note |
|---|
Updating this value will produce the loss of all persisted data and current pipelines. |
| Info |
|---|
This parameter should be removed if it is not used. |
|
<event_fetch_limit_in_items>
| int
| Optional
| Minimum value: 1
Maximum value: 1000 | This parameter controls the collector’s pagination size when the events are fetched. The default value is 100. | Info |
|---|
This parameter should be removed if it is not used. |
|
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Collector Docker image | SHA-256 hash |
|---|
collector-trend_micro_vision_one_collector_if-docker-image-1.0.0 | bf8485f7bf0a2374bf7b97b3a4a755e0b4c6fd495791e22a3077b479e4caa592
|
Use the following command to add the Docker image to the system:
| Code Block |
|---|
gunzip -c <image_file>-<version>.tgz | docker load |
| Note |
|---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value. |
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
| Code Block |
|---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
| Note |
|---|
Replace <product_name>, <image_name> and <version> with the proper values. |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.
| Code Block |
|---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:
| Code Block |
|---|
IMAGE_VERSION=<version> docker-compose up -d |
| Note |
|---|
Replace <product_name>, <image_name> and <version> with the proper values. |
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Events service
...
| title | Verify data collection |
|---|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
...
Component
...
Description
...
Setup
...
The setup module is in charge of authenticating the service and managing the token expiration when needed.
...
Puller
...
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
| Code Block |
|---|
INFO MainThread -> [SETUP] ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Puller Setup Started
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> We do not have a token. Getting a new one from the server.
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Attempting to get OAuth2 token from ThreatQuotient server....
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Attempting to get from client_id ThreatQuotient server....
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Successfully received a client_id token from (...)/assets/js/config.js
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Successfully received JWT token from (...) which expires in 3599 seconds
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Puller Setup Terminated
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Setup for module "ThreatQuotientDataPuller" has been successfully executed
|
Puller output
A successful initial run has the following output messages for the puller module:
| Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO MainThread -> [INPUT] ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Waiting until setup will be executed
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> PrePull Started
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> PrePull terminated
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Starting data collection every 5 seconds
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Pull Started. Retrieving timestamp: 2022-06-28 13:00:59.276966+00:00
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started getting events from ThreatQuotient
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started getting events from ThreatQuotient
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started sending events to Devo
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656421259.276966): Number of requests performed: 2; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 2 (from 1 unflattened events); Average of events per second: 4.179186813829765.
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> last_fetched_event_id and last_update_time saved in state: {'last_polled_time': 1656342870.739774, 'reset_persistence_auth': '', 'all_events_ids': [17653]}
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Last polled time saved in state: {'last_polled_time': 1656421259.276966, 'reset_persistence_auth': '', 'all_events_ids': [17653]}
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Pull terminated
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Data collection completed. Elapsed time: 0.483 seconds. Waiting for 4.517 second(s) until the next one
|
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
| Code Block |
|---|
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1655983326.290848): Number of requests performed: 2; Number of events received: 52; Number of duplicated events filtered out: 0; Number of events generated and sent: 52 (from 52 unflattened events); Average of events per second: 92.99414315733.
|
| Info |
|---|
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window. |
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps: Edit the configuration file. Change the value of the reset_persistence_auth parameter to a different one. Save the changes. Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided. | Note |
|---|
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events. |
|
...
time_in_utc>
collections:
- IP:
lookup_name: <socradar_custom_lookup_name>
include_collections: []
exclude_collections: []
- Domain:
lookup_name: <socradar_custom_lookup_name>
include_collections: [ ]
exclude_collections: [ ]
- URL:
lookup_name: <socradar_custom_lookup_name>
include_collections: [ ]
exclude_collections: [ ]
- Hash:
lookup_name: <socradar_custom_lookup_name>
include_collections: [ ]
exclude_collections: [ ] |
| Info |
|---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Replace the placeholders with your required values following the description table below: Parameter | Data type | Type | Value range / Format | Details |
|---|
<debug_status>
| bool
| Mandatory
| false / true
| This will make the collector generate (or not) log messages with the DEBUG level. | collector_id
| int | mandatory | minimum length: 1 maximum length: 5 | Use this parameter to give a unique ID to this collector. | collector_name
| str | mandatory | minimum length: 1 maximum length: 10 | Use this parameter to give a valid name to this collector. | devo_address
| str | mandatory | collector-us.devo.io
collector-eu.devo.io
| Use this parameter to identify the Devo Cloud where the events will be sent. | chain_filename
| str | mandatory | minimum length: 4 maximum length: 20 | Use this parameter to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt. | cert_filename
| str | mandatory | minimum length: 4 maximum length: 20 | Use this parameter to identify the file.cert downloaded from your Devo domain. | key_filename
| str | mandatory | minimum length: 4 maximum length: 20 | Use this parameter to identify the file.key downloaded from your Devo domain. | input_id
| int | mandatory | minimum length: 1 maximum length: 5 | Use this parameter to give a unique ID to this input service. This parameter is used to build the persistence address; do not use the same value for multiple collectors. It could cause a collision. | api_key
| str | mandatory | | The key obtained from SOCRadar for authentication. | company_id
| str | mandatory | | A company identification given by SOCRadar. | request_per_minute
| int | mandatory | | A rate limit value. A value of -1 means no restriction (not recommended). A typical value is 4 or 16, depending of the contracted plan with SOCRadar.
See “API limitations” section. | initial_start_time_in_utc
| str | optional | UTC datetime string | This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events. | tag
| str | optional | Devo tag-friendly string (no special characters, spaces, etc.) | An optional tag that allows users to override the service default tags. (see predefined tag names above) | socradar_custom_lookup_name
| str | optional | Valid name for a lookup (no special characters, no spaces) | Optional name for the lookup.. See the predefinied list of names above. |
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: Use the following command to add the Docker image to the system: | Code Block |
|---|
gunzip -c <image_file>-<version>.tgz | docker load |
| Note |
|---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value. |
The Docker image can be deployed on the following services: DockerExecute the following command on the root directory <any_directory>/devo-collectors/<product_name>/ | Code Block |
|---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
| Note |
|---|
Replace <product_name>, <image_name> and <version> with the proper values. |
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory. | Code Block |
|---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory: | Code Block |
|---|
IMAGE_VERSION=<version> docker-compose up -d |
| Note |
|---|
Replace <product_name>, <image_name> and <version> with the proper values. |
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration. |
API limitations
SOCRadar collector has some API limitations according to the contract.
...
This value should be introduced in the request_per_minute value described above.
If this limit is exceeded, a 429 error is returned by the SOCRadar API. In the event the collector encounters a 429 (that is, the requests hit a rate limit), the collector will attempt to gracefully respect the 429 rate limit and re-attempt the request three times before failing.
| Info |
|---|
The value of 4 request for minute seems too low for the collector to run with all the services enabled. Change the configuration to disable some services. |
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Events service
| Expand |
|---|
| title | Verify data collection |
|---|
|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console. This service has the following components: Component | Description |
|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. | Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
INFO InputProcess::SOCRadarPullerSetup(SOCRadar_collector,socradar#17812,audit_logs#predefined) -> Testing access to SOCRadar API for audit_logs
INFO InputProcess::SOCRadarPullerSetup(SOCRadar_collector,socradar#17812,audit_logs#predefined) -> Access to SOCRadar is OK
INFO InputProcess::SOCRadarPullerSetup(SOCRadar_collector,socradar#17812,audit_logs#predefined) -> Setup for module <SOCRadarPuller> has been successfully executed |
Puller outputA successful initial run has the following output messages for the puller module: | Info |
|---|
Note that the PrePull action is executed only one time before the first run of the Pull action. |
| Code Block |
|---|
INFO InputProcess::SOCRadarPuller(socradar,17812,audit_logs,predefined) -> Detected initial start time in UTC change: {'type_changes': {'root': {'old_type': <class 'pendulum.datetime.DateTime'>, 'new_type': <class 'NoneType'>, 'old_value': DateTime(2023, 1, 20, 0, 0, 0, tzinfo=Timezone('UTC')), 'new_value': None}}}. Setting last run time to 2023-01-20T00:00:00+00:00
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,IP) -> [pre_pull] -> Creating a new lookup_job_factory to be used by the pull method
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,IP) -> Detected initial start time in UTC change: {'type_changes': {'root': {'old_type': <class 'pendulum.datetime.DateTime'>, 'new_type': <class 'NoneType'>, 'old_value': DateTime(2023, 1, 20, 0, 0, 0, tzinfo=Timezone('UTC')), 'new_value': None}}}. Setting last run time to 2023-01-20T00:00:00+00:00
INFO InputProcess::SOCRadarPuller(socradar,17812,audit_logs,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,IP) -> Starting data collection every 3600 seconds
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Domain) -> [pre_pull] -> Creating a new lookup_job_factory to be used by the pull method
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,URL) -> [pre_pull] -> Creating a new lookup_job_factory to be used by the pull method
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Domain) -> Detected initial start time in UTC change: {'type_changes': {'root': {'old_type': <class 'pendulum.datetime.DateTime'>, 'new_type': <class 'NoneType'>, 'old_value': DateTime(2023, 1, 20, 0, 0, 0, tzinfo=Timezone('UTC')), 'new_value': None}}}. Setting last run time to 2023-01-20T00:00:00+00:00
INFO InputProcess::SOCRadarPuller(socradar,17812,audit_logs,predefined) -> Retrieving audit_logs events since 2023-01-20T00:00:00+00:00 to 2023-01-25 12:05:14.444890+00:00
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,URL) -> Detected initial start time in UTC change: {'type_changes': {'root': {'old_type': <class 'pendulum.datetime.DateTime'>, 'new_type': <class 'NoneType'>, 'old_value': DateTime(2023, 1, 20, 0, 0, 0, tzinfo=Timezone('UTC')), 'new_value': None}}}. Setting last run time to 2023-01-20T00:00:00+00:00
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,IP) -> Retrieving threat_feed events
INFO InputProcess::SOCRadarPuller(socradar,17812,audit_logs,predefined) -> Query range exceeds 1 day. Chunking into 6 query windows
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Hash) -> [pre_pull] -> Creating a new lookup_job_factory to be used by the pull method
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Domain) -> Starting data collection every 3600 seconds
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,IP) -> Retrieving IP indicators from SOCRadar
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,URL) -> Starting data collection every 3600 seconds
INFO InputProcess::SOCRadarPuller(socradar,17812,audit_logs,predefined) -> Querying window 1 of 6 (2023-01-20T00:00:00+00:00 - 2023-01-20T23:59:59.999999+00:00)
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Hash) -> Detected initial start time in UTC change: {'type_changes': {'root': {'old_type': <class 'pendulum.datetime.DateTime'>, 'new_type': <class 'NoneType'>, 'old_value': DateTime(2023, 1, 20, 0, 0, 0, tzinfo=Timezone('UTC')), 'new_value': None}}}. Setting last run time to 2023-01-20T00:00:00+00:00
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Domain) -> Retrieving threat_feed events
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,URL) -> Retrieving threat_feed events
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Domain) -> Retrieving Domain indicators from SOCRadar
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Hash) -> Starting data collection every 3600 seconds
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,URL) -> Retrieving URL indicators from SOCRadar
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Hash) -> Retrieving threat_feed events
INFO InputProcess::SOCRadarThreatIntelligencePuller(socradar,17812,threat_feed,predefined,Hash) -> Retrieving Hash indicators from SOCRadar |
After a successful collector’s execution (that is, no error logs found), you will see the following log message: | Code Block |
|---|
INFO InputProcess::SOCRadarPuller(socradar,17812,audit_logs,predefined) -> Retrieving audit_logs events since 2023-01-25T13:05:14.546952+00:00 to 2023-01-25 13:06:14.547341+00:00 |
| Info |
|---|
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window. |
|
| Expand |
|---|
| title | Restart the persistence |
|---|
|
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps: Edit the configuration file. Change the value of the initial_start_time_in_utc parameter to a different one. Save the changes. Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided. | Note |
|---|
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events. |
|
| Expand |
|---|
|
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors. Error type | Error ID | Error message | Cause | Solution |
|---|
SetupError | 101 | Error during auth check. Please doublecheck your credentials: [..]
| SOCRadar reject the credentials. | Obtain new api_key from SOCRadar. | InitVariablesError | 1 | Error while fetching collector variables: [..]
| One of required parameters is absent or a parameter has an incorrect type | Add the parameter or correct the type. | 2 | Error validating collector variables: [...]
| One of the parameters has the correct type but it is not correct. For instance, a date is not a valid date, or a number is out of its range. | Correct the parameter in the configuration file. | 3 | Error while creating client: [..]
| Other error inizializing the collector. | Solution depends on specific error message. | Prepull Error | 200 | Error during pre_pull: [...]
| Something was wrong creating the checkpoints | Review the initial_start_time_in_utc | Pull Error | 300 | Encountered pull error: [...]
| Something was wrong contacting the API. | Read the HTTP error code as long as the response’s text. This information should be enough to understand why is the error happening. |
|
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
| Expand |
|---|
| title | Verify collector operations |
|---|
|
InitializationThe initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module: | Code Block |
|---|
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> List of files in directory "/devo-collector/config": []
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "/devo-collector/config/config.yaml", "config_validated": False, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": True, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": True}
2023-01-25T12:54:13.436 INFO MainProcess::MainThread -> Build time: "2023-01-24T17:01:02.855958522+0000", OS: "Linux-5.4.149-73.259.amzn2.x86_64-x86_64-with-glibc2.31", collector(name:version): "SOCRadar_collector:1.0.0", owner: "integrations_factory@devo.com", started at: "2023-01-25T12:54:13.433352Z"
2023-01-25T12:54:13.440 INFO MainProcess::MainThread -> Initialized all object from "MainProcess" process
2023-01-25T12:54:13.440 INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=120s)
2023-01-25T12:54:13.443 INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=120s)
2023-01-25T12:54:13.444 INFO OutputProcess::MainThread -> Process started
2023-01-25T12:54:13.448 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-01-25T12:54:13.448 INFO InputProcess::MainThread -> Process Started
2023-01-25T12:54:13.449 INFO MainProcess::CollectorThread -> global_status: {"main_process": {"process_id": 7, "process_status": "sleeping", "thread_counter": 5, "thread_names": ["MainThread", "QueueFeederThread", "OutputControllerThread", "InputControllerThread", "CollectorThread"], "memory_info": {"rss": "51.41MiB", "vms": "360.62MiB", "shared": "14.38MiB", "text": "8.00KiB", "lib": "0.00B", "data": "76.79MiB", "dirty": "0.00B"}, "CollectorThread": {"running_flag": true, "status": "RUNNING(10)", "shutdown_timestamp": "None", "message_queues": {"standard": {"name": "standard_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7f69f5458370>"}, "lookup": {"name": "lookup_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7f69f5472fd0>"}, "internal": {"name": "internal_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 2, "put_lock": "<Lock(owner=SomeOtherProcess)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7f69f54ca9a0>"}}, "controllers": {"InputControllerThread": {"running_flag": true, "status": "RUNNING(10)", "underlying_process": {"process_id": 12, "process_status": "running", "num_threads": 2, "memory_info": {"rss": "41.45MiB", "vms": "212.62MiB", "shared": "4.45MiB", "text": "8.00KiB", "lib": "0.00B", "data": "56.54MiB", "dirty": "0.00B"}, "exit_code": null}}, "OutputControllerThread": {"running_flag": true, "status": "RUNNING(10)", "underlying_process": {"process_id": 10, "process_status": "running", "num_threads": 1, "memory_info": {"rss": "40.77MiB", "vms": "138.61MiB", "shared": "3.79MiB", "text": "8.00KiB", "lib": "0.00B", "data": "46.41MiB", "dirty": "0.00B"}, "exit_code": null}}}}}}
2023-01-25T12:54:13.451 INFO MainProcess::CollectorThread -> environment_status: {"cpu_count": 8, "cpu_usage": "1:3.09, 5:3.26, 15:3.67, ", "virtual_memory": {"total": "62.03GiB", "available": "56.24GiB", "percent": 9.3, "used": "5.50GiB", "free": "43.61GiB", "active": "4.72GiB", "inactive": "11.08GiB", "buffers": "2.64MiB", "cached": "12.92GiB", "shared": "9.35MiB", "slab": "2.19GiB"}, "disk_usage": {"/dev/termination-log": 5.3, "/etc/hosts": 5.3, "/etc/resolv.conf": 5.3, "/etc/hostname": 5.3, "/etc/devo/collector": 5.3}, "processes_info": [{"pid": 1, "ppid": 0, "process_name": "sh", "process_cmd": "/bin/sh -c devo-collector --config ${CONFIG_FILE:-config.yaml} --prod-mode", "memory_usage": {"rss": "596.00KiB", "vms": "2.55MiB", "shared": "528.00KiB", "text": "96.00KiB", "lib": "0.00B", "data": "316.00KiB", "dirty": "0.00B"}}, {"pid": 7, "ppid": 1, "process_name": "devo-collector", "process_cmd": "/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode", "memory_usage": {"rss": "51.57MiB", "vms": "360.62MiB", "shared": "14.49MiB", "text": "8.00KiB", "lib": "0.00B", "data": "76.79MiB", "dirty": "0.00B"}}, {"pid": 10, "ppid": 7, "process_name": "devo-collector", "process_cmd": "/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode", "memory_usage": {"rss": "40.77MiB", "vms": "138.61MiB", "shared": "3.79MiB", "text": "8.00KiB", "lib": "0.00B", "data": "46.41MiB", "dirty": "0.00B"}}, {"pid": 12, "ppid": 7, "process_name": "devo-collector", "process_cmd": "/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode", "memory_usage": {"rss": "41.46MiB", "vms": "212.62MiB", "shared": "4.45MiB", "text": "8.00KiB", "lib": "0.00B", "data": "56.54MiB", "dirty": "0.00B"}}]}
2023-01-25T12:54:13.493 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The <address> is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io
2023-01-25T12:54:13.498 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The <address> is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io
2023-01-25T12:54:13.502 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The <address> is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io
2023-01-25T12:54:13.503 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,relay_0) -> Starting thread (every 300 seconds)
2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,relay_0) -> Starting thread
2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread |
Events delivery and Devo ingestionThe event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module: | Code Block |
|---|
2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Added "/devo-collector" directory to the Python path
2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Added "/devo-collector/config_internal" directory to the Python path
2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Added "/devo-collector/schemas" directory to the Python path
2023-01-25T12:54:13.396 INFO MainProcess::MainThread -> Production mode: True, execute only setup and exit: False, Python version: "3.9.10 (main, Jan 24 2023, 13:13:38) [GCC 9.4.0]", current dir: "/devo-collector", exists "config" dir: True, exists "config_internal" dir: True, exists "certs" dir: True, exists "schemas" dir: True, exists "credentials" dir: True
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> List of files in directory "/devo-collector/config": []
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-01-25T12:54:13.419 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "/devo-collector/config/config.yaml", "config_validated": False, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": True, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": True}
2023-01-25T12:54:13.436 INFO MainProcess::MainThread -> Build time: "2023-01-24T17:01:02.855958522+0000", OS: "Linux-5.4.149-73.259.amzn2.x86_64-x86_64-with-glibc2.31", collector(name:version): "SOCRadar_collector:1.0.0", owner: "integrations_factory@devo.com", started at: "2023-01-25T12:54:13.433352Z"
2023-01-25T12:54:13.440 INFO MainProcess::MainThread -> Initialized all object from "MainProcess" process
2023-01-25T12:54:13.440 INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=120s)
2023-01-25T12:54:13.443 INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=120s)
2023-01-25T12:54:13.444 INFO OutputProcess::MainThread -> Process started
2023-01-25T12:54:13.448 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-01-25T12:54:13.448 INFO InputProcess::MainThread -> Process Started
2023-01-25T12:54:13.449 INFO MainProcess::CollectorThread -> global_status: {"main_process": {"process_id": 7, "process_status": "sleeping", "thread_counter": 5, "thread_names": ["MainThread", "QueueFeederThread", "OutputControllerThread", "InputControllerThread", "CollectorThread"], "memory_info": {"rss": "51.41MiB", "vms": "360.62MiB", "shared": "14.38MiB", "text": "8.00KiB", "lib": "0.00B", "data": "76.79MiB", "dirty": "0.00B"}, "CollectorThread": {"running_flag": true, "status": "RUNNING(10)", "shutdown_timestamp": "None", "message_queues": {"standard": {"name": "standard_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7f69f5458370>"}, "lookup": {"name": "lookup_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=None)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7f69f5472fd0>"}, "internal": {"name": "internal_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 2, "put_lock": "<Lock(owner=SomeOtherProcess)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x7f69f54ca9a0>"}}, "controllers": {"InputControllerThread": {"running_flag": true, "status": "RUNNING(10)", "underlying_process": {"process_id": 12, "process_status": "running", "num_threads": 2, "memory_info": {"rss": "41.45MiB", "vms": "212.62MiB", "shared": "4.45MiB", "text": "8.00KiB", "lib": "0.00B", "data": "56.54MiB", "dirty": "0.00B"}, "exit_code": null}}, "OutputControllerThread": {"running_flag": true, "status": "RUNNING(10)", "underlying_process": {"process_id": 10, "process_status": "running", "num_threads": 1, "memory_info": {"rss": "40.77MiB", "vms": "138.61MiB", "shared": "3.79MiB", "text": "8.00KiB", "lib": "0.00B", "data": "46.41MiB", "dirty": "0.00B"}, "exit_code": null}}}}}}
2023-01-25T12:54:13.451 INFO MainProcess::CollectorThread -> environment_status: {"cpu_count": 8, "cpu_usage": "1:3.09, 5:3.26, 15:3.67, ", "virtual_memory": {"total": "62.03GiB", "available": "56.24GiB", "percent": 9.3, "used": "5.50GiB", "free": "43.61GiB", "active": "4.72GiB", "inactive": "11.08GiB", "buffers": "2.64MiB", "cached": "12.92GiB", "shared": "9.35MiB", "slab": "2.19GiB"}, "disk_usage": {"/dev/termination-log": 5.3, "/etc/hosts": 5.3, "/etc/resolv.conf": 5.3, "/etc/hostname": 5.3, "/etc/devo/collector": 5.3}, "processes_info": [{"pid": 1, "ppid": 0, "process_name": "sh", "process_cmd": "/bin/sh -c devo-collector --config ${CONFIG_FILE:-config.yaml} --prod-mode", "memory_usage": {"rss": "596.00KiB", "vms": "2.55MiB", "shared": "528.00KiB", "text": "96.00KiB", "lib": "0.00B", "data": "316.00KiB", "dirty": "0.00B"}}, {"pid": 7, "ppid": 1, "process_name": "devo-collector", "process_cmd": "/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode", "memory_usage": {"rss": "51.57MiB", "vms": "360.62MiB", "shared": "14.49MiB", "text": "8.00KiB", "lib": "0.00B", "data": "76.79MiB", "dirty": "0.00B"}}, {"pid": 10, "ppid": 7, "process_name": "devo-collector", "process_cmd": "/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode", "memory_usage": {"rss": "40.77MiB", "vms": "138.61MiB", "shared": "3.79MiB", "text": "8.00KiB", "lib": "0.00B", "data": "46.41MiB", "dirty": "0.00B"}}, {"pid": 12, "ppid": 7, "process_name": "devo-collector", "process_cmd": "/usr/local/bin/python /usr/local/bin/devo-collector --config config.yaml --prod-mode", "memory_usage": {"rss": "41.46MiB", "vms": "212.62MiB", "shared": "4.45MiB", "text": "8.00KiB", "lib": "0.00B", "data": "56.54MiB", "dirty": "0.00B"}}]}
2023-01-25T12:54:13.493 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The <address> is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io
2023-01-25T12:54:13.498 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The <address> is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io
2023-01-25T12:54:13.502 WARNING OutputProcess::MainThread -> DevoSender::_validate_kwargs_for_method__init__ -> The <address> is not a standard address (collector-eu.devo.io, collector-us.devo.io): collector-ap.devo.io
2023-01-25T12:54:13.503 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,relay_0) -> Starting thread (every 300 seconds)
2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,relay_0) -> Starting thread
2023-01-25T12:54:13.504 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-01-25T12:54:13.505 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,relay_0) -> Starting thread (every 300 seconds)
2023-01-25T12:54:13.505 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,relay_0) -> Starting thread
2023-01-25T12:54:13.505 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-01-25T12:54:13.506 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,relay_0) -> Starting thread (every 300 seconds)
2023-01-25T12:54:13.506 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,relay_0) -> Starting thread
2023-01-25T12:54:13.526 INFO InputProcess::MainThread -> InputThread(socradar,17812) - Starting thread (execution_period=60s)
2023-01-25T12:54:13.526 INFO InputProcess::MainThread -> ServiceThread(socradar,17812,incidents,predefined) - Starting thread (execution_period=60s)
2023-01-25T12:54:13.526 INFO InputProcess::MainThread -> SOCRadarPullerSetup(SOCRadar_collector,socradar#17812,incidents#predefined) -> Starting thread
2023-01-25T12:54:13.527 INFO InputProcess::MainThread -> SOCRadarPuller(socradar,17812,incidents,predefined) - Starting thread |
| Info |
|---|
By default, these information traces will be displayed every 10 minutes. |
Sender servicesThe Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services: Sender services | Description |
|---|
internal_senders
| In charge of delivering internal metrics to Devo such as logging traces or metrics. | standard_senders
| In charge of delivering pulled events to Devo. |
Sender statisticsEach service displays its own performance statistics that allow checking how many events have been delivered to Devo by type: Logging trace | Description |
|---|
Number of available senders: 1
| Displays the number of concurrent senders available for the given Sender Service. | sender manager internal queue size: 0
| Displays the items available in the internal sender queue. | Info |
|---|
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
| Standard - Total number of messages sent: 57, messages sent since "2023-01-10 16:09:16.116750+00:00": 0 (elapsed 0.000 seconds
| Displayes the number of events from the last time and following the given example, the following conclusions can be obtained: 44 events were sent to Devo since the collector started. The last checkpoint timestamp was 2023-01-10 16:09:16.116750+00:00. 21 events where sent to Devo between the last UTC checkpoint and now. Those 21 events required 0.00 seconds to be delivered.
| Info |
|---|
By default these traces will be shown every 10 minutes. |
|
|
