Document toolboxDocument toolbox

Absolute collector

Devo collector features

Feature

Details

Feature

Details

Allow parallel downloading (multipod)

not allowed

Running environments

  • collector server

  • on-premise

Populated Devo events

table

Flattening preprocessing

no

Requires IP Whitelisting

yes

Data sources

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

event

Get all the siem event type.

/v3/reporting/siem-events

siem_events

siem.absolute.reporting.event

v1.0.0

For more information on how the events are parsed, visit our page.

Flattening preprocessing

Data source

Collector service

Optional

Flattening details

Data source

Collector service

Optional

Flattening details

event

siem_events

yes

not required

Accepted authentication methods

Authentication method

token_id

token_secret

Authentication method

token_id

token_secret

token_id

required

required

token_secret

required

required

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

Setting

Details

token_id

The Token id for absolute API.

token_secret

The Token secret for absolute API.

See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Siem events

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in an organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2024-10-24T09:46:34.464 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread 2024-10-24T09:46:34.465 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "vendor_requests" created: "Number of requests received from the vendor API", unit: "requests" 2024-10-24T09:46:34.465 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_2) -> Starting thread (every 300 seconds) 2024-10-24T09:46:34.466 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_received" created: "Number of messages received from the vendor API", unit: "1" 2024-10-24T09:46:34.466 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_removed" created: "Number of messages removed by the collector", unit: "1" 2024-10-24T09:46:34.467 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_filtered" created: "Number of messages filtered by the collector", unit: "1" 2024-10-24T09:46:34.467 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-10-24T09:46:34.467 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01 2024-10-24T09:46:34.467 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_counter" created: "Number of messages enqueued", unit: "1" 2024-10-24T09:46:34.467 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> Starting thread 2024-10-24T09:46:34.467 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_bytes" created: "Number of bytes enqueued", unit: "1" 2024-10-24T09:46:34.468 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_counter" created: "Number of messages enqueued", unit: "1" 2024-10-24T09:46:34.468 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-10-24T09:46:34.468 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_bytes" created: "Number of messages enqueued", unit: "1" 2024-10-24T09:46:34.468 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-10-24T09:46:34.469 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-10-24T09:46:34.469 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_counter" created: "Number of messages enqueued in the queue", unit: "1" 2024-10-24T09:46:34.469 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_bytes" created: "Number of messages enqueued in the queue", unit: "1" 2024-10-24T09:46:34.469 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-10-24T09:46:34.470 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Gauge "module_global_status" created: "Global status of current module", unit: "1" 2024-10-24T09:46:34.471 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-10-24T09:46:34.471 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-10-24T09:46:34.472 INFO OutputProcess::MainThread -> OutputMetricsThread -> Started thread for updating metrics values (update_period=10.0) 2024-10-24T09:46:34.473 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-10-24T09:46:34.473 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-10-24T09:46:34.512 INFO OutputProcess::MainThread -> [GC] global: 87.8% -> 87.8%, process: RSS(62.86MiB -> 62.98MiB), VMS(1.07GiB -> 1.07GiB) 2024-10-24T09:46:34.514 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_counter" created: "Number of messages sent to the defined output", unit: "1" 2024-10-24T09:46:34.517 INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_bytes" created: "Number of bytes sent to the defined output", unit: "1" 2024-10-24T09:46:34.521 WARNING MainProcess::CollectorThread -> There is a process that now it doesn't exists (pid=1197409) 2024-10-24T09:46:34.927 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/pulkit/devo/collectors/devo-collector-absolute/certs/chain.crt", "cert_path": "/home/pulkit/devo/collectors/devo-collector-absolute/certs/int-if-integrations-india.crt", "key_path": "/home/pulkit/devo/collectors/devo-collector-absolute/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-APAC-0049", session_id: "125162671458480" 2024-10-24T09:46:34.928 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-10-24T09:46:35.582 INFO InputProcess::AbsolutePullerSetup(absolute#74383,siem_events#predefined) -> Setup for module <AbsolutePuller> has been successfully executed 2024-10-24T09:46:36.358 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> AbsolutePuller(absolute#74383,siem_events#predefined) Starting the execution of pre_pull() 2024-10-24T09:46:36.360 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Reading persisted data 2024-10-24T09:46:36.361 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Data retrieved from the persistence: None 2024-10-24T09:46:36.362 WARNING InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Persistence will be overridden due to the retrieved state is empty 2024-10-24T09:46:36.362 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Running the persistence upgrade steps 2024-10-24T09:46:36.363 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Running the persistence corrections steps 2024-10-24T09:46:36.363 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Running the persistence corrections steps 2024-10-24T09:46:36.363 WARNING InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Some changes have been detected and the persistence needs to be updated. Previous content: None. New content: {'@persistence_version': 1, 'initial_start_time_in_utc': '2024-10-21T10:15:30.000Z', 'last_event_time_in_utc': '2024-10-21T10:15:30.000Z', 'last_ids': []} 2024-10-24T09:46:36.365 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Updating the persistence 2024-10-24T09:46:36.366 WARNING InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Persistence has been updated successfully 2024-10-24T09:46:36.366 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> AbsolutePuller(absolute#74383,siem_events#predefined) Finalizing the execution of pre_pull() 2024-10-24T09:46:36.367 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Starting data collection every 300 seconds 2024-10-24T09:46:36.367 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Pull Started 2024-10-24T09:46:36.368 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Fetching data From : 2024-10-21T10:15:30.000Z to 2024-10-24T04:16:36.358Z 2024-10-24T09:46:38.465 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1729743396358):Number of requests made: 1; Number of events received: 1000; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000; Average of events per second: 476.757. 2024-10-24T09:46:38.467 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Sent 1000 siem_events events to Devo.

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-10-24T09:46:36.367 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Pull Started 2024-10-24T09:46:36.368 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Fetching data From : 2024-10-21T10:15:30.000Z to 2024-10-24T04:16:36.358Z 2024-10-24T09:46:38.465 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1729743396358):Number of requests made: 1; Number of events received: 1000; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000; Average of events per second: 476.757. 2024-10-24T09:46:38.467 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> Sent 1000 siem_events events to Devo.

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-10-24T09:46:38.465 INFO InputProcess::AbsolutePuller(absolute#74383,siem_events#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1729743396358):Number of requests made: 1; Number of events received: 1000; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000; Average of events per second: 476.757.

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

  1. Edit the configuration file.

  2. Change the value of the initial_start_time_in_utc_value parameter to a different one.

  3. Save the changes.

  4. Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Error Type

Error Id

Error Message

Cause

Solution

Error Type

Error Id

Error Message

Cause

Solution

InitVariablesError

1

initial_start_time_in_utc is not set as per the datetime_format : {datetime_format}

The date in config is not as per required format

Ensure the date format is correct.

InitVariablesError

2

initial_start_time_in_utc is either in the future or older than 72 hours"

The date in config is not in the required period

Ensure the date period is correct.

InitVariablesError

3

page_limit

The date in config is not in the required period

Ensure the date period is correct.

SetupError

100

HTTP Error occurred while retrieving events from Absolute server

Absolute API call is failing

Check the credentials and ensure that the collector has the necessary permissions to access the Absolute API.

PullError

300

HTTP Error occurred while fetching siem event from absolute server

Absolute API call is failing

Check the credentials and ensure that the collector has the necessary permissions to access the Absolute API.

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

  • The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

  • The global pressure of the available memory is displayed in the global value.

  • All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

Change log

Release

Released on

Release type

Details

Recommendations

Release

Released on

Release type

Details

Recommendations

v1.0.0

Oct 30, 2024

NEW FEATURE

New collector

Recommended version