Overview
-
Devo collector features
Feature | Details |
|---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Allowed source events obfuscation |
|
Data sources
Data source | Description | API endpoint | Collector service name | Devo table |
|---|---|---|---|---|
Waf logs | Download log files from the Imperva storage repository |
|
|
|
For more information on how the events are parsed, visit our page.
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting | Details |
| The API ID |
| The API key |
| The API URL |
Accepted authentication methods
Authentication method | Token |
Basic auth | REQUIRED |
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Services (Services)
Internal process and deduplication method
The collector will use the ID category for deduplication. The Ids will be stored in redis, and checked against.
Devo categorization and destination
All services when pulling the event will check the workload and append that workload to the tag.
Setup/puller output
2024-09-27T15:07:10.727 INFO InputProcess::MainThread -> ImpervaCommonPullerSetup(imperva,imperva#73922,imperva_logs#predefined) -> Starting thread
2024-09-27T15:07:10.727 INFO InputProcess::MainThread -> ImpervaDataPuller(imperva#73922,imperva_logs#predefined) - Starting thread
2024-09-27T15:07:10.727 INFO InputProcess::ImpervaCommonPullerSetup(imperva,imperva#73922,imperva_logs#predefined) -> Auth header created.
2024-09-27T15:07:10.732 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-27T15:07:10.732 WARNING InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Waiting until setup will be executed
2024-09-27T15:07:10.732 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/pulkit/devo/collectors/devo-collector-imperva/state/8812b7237c673c4ed40c4adb7f5eaf01"
2024-09-27T15:07:10.739 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/pulkit/devo/collectors/devo-collector-imperva/state/not_used/DevoSender;lookup_senders;devo_sender_0.json.gz
2024-09-27T15:07:10.743 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-27T15:07:10.743 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/pulkit/devo/collectors/devo-collector-imperva/state/df8895fef2a509cbd87fcc9850dc0c81"
2024-09-27T15:07:10.747 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/pulkit/devo/collectors/devo-collector-imperva/state/not_used/OutputLookupConsumer;lookup_senders;0.json.gz
2024-09-27T15:07:10.750 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-27T15:07:10.750 INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/pulkit/devo/collectors/devo-collector-imperva/state/865a79c1b99ad39b22becc235c9732cb"
2024-09-27T15:07:10.756 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/pulkit/devo/collectors/devo-collector-imperva/state/not_used/DevoSenderManager;internal_senders;devo_2.json.gz
2024-09-27T15:07:10.760 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-27T15:07:10.760 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/pulkit/devo/collectors/devo-collector-imperva/state/65227e92e07799e6e01b6ecd0bc0e76a"
2024-09-27T15:07:10.771 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/pulkit/devo/collectors/devo-collector-imperva/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz
2024-09-27T15:07:10.776 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-27T15:07:10.776 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/pulkit/devo/collectors/devo-collector-imperva/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-09-27T15:07:10.780 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/pulkit/devo/collectors/devo-collector-imperva/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-09-27T15:07:10.785 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-09-27T15:07:10.785 INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/pulkit/devo/collectors/devo-collector-imperva/state/10dd360c86621afd5a28a029a0dddcf6"
2024-09-27T15:07:10.787 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-09-27T15:07:10.788 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_2) -> Starting thread (every 300 seconds)
2024-09-27T15:07:10.789 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_2) -> Starting thread
2024-09-27T15:07:10.790 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-27T15:07:10.791 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-27T15:07:10.791 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-09-27T15:07:10.797 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_2) -> Starting thread (every 300 seconds)
2024-09-27T15:07:10.799 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_2) -> Starting thread
2024-09-27T15:07:10.800 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-27T15:07:10.801 INFO InputProcess::MainThread -> [GC] global: 68.9% -> 69.0%, process: RSS(43.85MiB -> 43.97MiB), VMS(1.42GiB -> 1.42GiB)
2024-09-27T15:07:10.801 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-27T15:07:10.801 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-27T15:07:10.802 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-27T15:07:10.804 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-09-27T15:07:10.805 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_2) -> Starting thread (every 300 seconds)
2024-09-27T15:07:10.806 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> Starting thread
2024-09-27T15:07:10.807 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-27T15:07:10.807 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-27T15:07:10.807 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-27T15:07:10.807 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-27T15:07:10.808 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-27T15:07:10.808 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-09-27T15:07:10.811 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-27T15:07:10.811 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-09-27T15:07:10.811 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-27T15:07:10.812 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.02
2024-09-27T15:07:10.814 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-27T15:07:10.815 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.01
2024-09-27T15:07:10.878 INFO OutputProcess::MainThread -> [GC] global: 69.0% -> 69.0%, process: RSS(43.20MiB -> 43.32MiB), VMS(1.91GiB -> 1.91GiB)
2024-09-27T15:07:11.465 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/pulkit/devo/collectors/devo-collector-imperva/certs/chain.crt", "cert_path": "/home/pulkit/devo/collectors/devo-collector-imperva/certs/int-if-integrations-india.crt", "key_path": "/home/pulkit/devo/collectors/devo-collector-imperva/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-APAC-0049", session_id: "130027055775408"
2024-09-27T15:07:11.466 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-09-27T15:07:12.772 INFO InputProcess::ImpervaCommonPullerSetup(imperva,imperva#73922,imperva_logs#predefined) -> successfully received details from Imperva server
2024-09-27T15:07:12.772 INFO InputProcess::ImpervaCommonPullerSetup(imperva,imperva#73922,imperva_logs#predefined) -> Api_id and Api_key given - so service auth is valid (completed)
2024-09-27T15:07:12.773 INFO InputProcess::ImpervaCommonPullerSetup(imperva,imperva#73922,imperva_logs#predefined) -> Api_id and Api_key given - so service auth is valid (completed)
2024-09-27T15:07:12.773 INFO InputProcess::ImpervaCommonPullerSetup(imperva,imperva#73922,imperva_logs#predefined) -> Setup for module <ImpervaDataPuller> has been successfully executed
2024-09-27T15:07:12.802 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> ImpervaDataPuller(imperva#73922,imperva_logs#predefined) Starting the execution of pre_pull()
2024-09-27T15:07:12.803 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Reading persisted data
2024-09-27T15:07:12.806 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Data retrieved from the persistence: {'id_last_pull': '', '@persistence_version': 1}
2024-09-27T15:07:12.806 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Running the persistence upgrade steps
2024-09-27T15:07:12.807 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Running the persistence corrections steps
2024-09-27T15:07:12.807 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Running the persistence corrections steps
2024-09-27T15:07:12.807 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> No changes were detected in the persistence
2024-09-27T15:07:12.808 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> ImpervaDataPuller(imperva#73922,imperva_logs#predefined) Finalizing the execution of pre_pull()
2024-09-27T15:07:12.808 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Starting data collection every 60 seconds
2024-09-27T15:07:12.809 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> Pull Started
2024-09-27T15:07:14.985 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> successfully received details from Imperva server
2024-09-27T15:07:18.278 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> successfully received details from Imperva server
2024-09-27T15:07:18.613 INFO InputProcess::ImpervaDataPuller(imperva#73922,imperva_logs#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1727429832802):Number of requests made: 1; Number of events received: 7007; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-09-27T15:07:18.868 INFO OutputProcess::DevoSender(standard_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(standard_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/pulkit/devo/collectors/devo-collector-imperva/certs/chain.crt", "cert_path": "/home/pulkit/devo/collectors/devo-collector-imperva/certs/int-if-integrations-india.crt", "key_path": "/home/pulkit/devo/collectors/devo-collector-imperva/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-APAC-0049", session_id: "130027054493360"
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
short_idto a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
InitVariablesError | 101 | Cannot run both | Can't have both | Pick one or the other |
PullError | 301 | Error in the request itself - bad date perhaps? : {response.json()} | The date could be in the wrong format | Check the credentials. |
PullError | 302 | access token has expired or invalid. status code: 401. Error message: {response.json()} | access token has expired or invalid | Check the credential. |
PullError | 303 | The access token does not have valid permissions to perform this request. Add the required status code: 403. Error message: {response.json()} | The access token does not have valid permissions to perform this request | Check the credential. |
PullError | 304 | The resource requested is not found. Resource Url: {request_url} status code: 404. Error message: {response.json()} | The resource requested is not found. | Read Stack trace and status code to determine |
PullError | 305 | Unexpected error occurred at the Imperva server. status code: {response.status_code}. Error message: {response.json()} | Unexpected error occurred at the Imperva server. | Read Stack trace and status code to determine |
SetupError | 100 | Unexpected error occurred at the Imperva server. status code: {response.status_code}. Error message: {response.json()} | Unexpected error occurred at the Imperva server. | Read Stack trace and status code to determine |
Collector operations
Verify collector operations
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
2024-09-27T15:07:10.586 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2024-09-27T15:07:10.586 INFO MainProcess::MainThread -> "/etc/devo/job" does not exists
2024-09-27T15:07:10.586 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2024-09-27T15:07:10.586 INFO MainProcess::MainThread -> "/etc/devo/collector" does not exists
2024-09-27T15:07:10.586 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "/home/pulkit/devo/collectors/devo-collector-imperva/config/local_config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
2024-09-27T15:07:10.591 WARNING MainProcess::MainThread -> [WARNING] Illegal global setting has been ignored -> multiprocessing: False
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module:
2024-09-27T15:22:38.030 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread 2024-09-27T15:22:38.030 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_2) -> Starting thread (every 300 seconds) 2024-09-27T15:22:38.031 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_2) -> Starting thread 2024-09-27T15:22:38.031 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-09-27T15:22:38.032 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
Sender services
The Integrations Factory Collector SDK has 3 different sender services depending on the event type to deliver (internal, standard, and lookup). This collector uses the following Sender Services:
Logging trace | Description |
Number of available senders: 1 | Displays the number of concurrent senders available for the given Sender Service. |
Sender manager internal queue size: 0 | Displays the items available in the internal sender queue. This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds) | Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained:
|
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace | Description |
Number of available senders: 1 | Displays the number of concurrent senders available for the given Sender Service |
Sender manager internal queue size: 0 | Displays the items available in the internal sender queue. |
Standard - Total number of messages sent: 57, messages sent since "2023-01-10 16:09:16.116750+00:00": 0 (elapsed 0.000 seconds | Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained:
|
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
2024-09-27T15:24:38.090 INFO InputProcess::MainThread -> [GC] global: 69.2% -> 69.2%, process: RSS(227.16MiB -> 227.16MiB), VMS(1.55GiB -> 1.55GiB) 2024-09-27T15:24:38.104 INFO OutputProcess::MainThread -> [GC] global: 69.2% -> 69.2%, process: RSS(66.07MiB -> 66.07MiB), VMS(1.91GiB -> 1.91GiB)
Change log
Release | Released on | Release type | Details | Recommendations |
|---|---|---|---|---|
| NEW COLLECTOR | New collector |
|