Darktrace Respond collector

Overview

Darktrace RESPOND works autonomously to disarm attacks whenever they occur. Reacts to threats in seconds, working 24/7 as it frees up security teams and resources.

Darktrace Self-Learning AI delivers precise information about what’s not normal to your organization. Darktrace RESPOND takes precise action to neutralize threats against any and every asset, no matter where data resides.

Devo collector features

Feature	Details
Allow parallel downloading (`multipod`)	`not allowed`
Running environments	`collector server` `on-premise`
Populated Devo events	`table`
Flattening preprocessing	`yes`
Allowed source events obfuscation	`yes`

Data sources

Data source	Description	API endpoint	Collector service name	Devo table	Available from release
Antigena Actions	Gives information about current and past Darktrace RESPOND/Network (formerly Antigena Network) actions.	`/antigena`	`antigena`	`edr.darktrace.respond.antigena`	`v1.0.0`
AiAnalyst Incident Events	Provides access to AI Analyst events - a group of anomalies or network activity investigated by Cyber AI Analyst.	`/aianalyst/incidentevents`	`aianalyst_incidentevents`	`edr.darktrace.respond.incident_event`	`v1.0.0`
Summary Statistics	Returns simple statistics on device counts, processed bandwidth and the number of active Darktrace RESPOND actions.	`/summarystatistics`	`summarystatistics`	`edr.darktrace.respond.summary`	`v1.0.0`
Status	Detailed system health information from the Status page .	`/status`	`status`	`edr.darktrace.respond.status`	`v1.0.0`
Modelbreaches	Returns a time-sorted list of model breaches, filtered by the specified parameters.	`/modelbreaches`	`modelbreaches`	`edr.darktrace.respond.model_breach`	`v1.0.1`

For more information on how the events are parsed, visit our page.

Flattening preprocessing

Data source

Collector service

Optional

Flattening details

Status

status

Yes

Original :

"time":"2023-05-31 09:37",
   "installed":"2022-06-15",
   "mobileAppConfigured":false,
   "version":"6.0.32 (a1c388)",
   "ipAddress":"172.27.24.26",
   "modelsUpdated":"2023-05-30 18:46:46",
   "modelPackageVersion":"6.0.23-1019~20230530170515~g6d5204",
   "bundleVersion":"60076",
"instances":{
      "1":{
         "id":1,
         "downCount":1,
         "upCount":1,
         "downTimeMs":1685514150000000,
         "downTime":"2023-05-31 06:22:30",
         "version":"6.0.32 (a1c388)",
         "ipAddress":"172.27.24.26",

     }
"2":{
         "id":2,
         "downCount":8,
         "upCount":8,
         "downTimeMs":1685516996000000,
         "downTime":"2023-05-31 07:09:56",
         "version":"6.0.32 (a1c388)",
         "ipAddress":"10.32.1.99",
     }
}

Result:

"time":"2023-05-31 09:37",
   "installed":"2022-06-15",
   "mobileAppConfigured":false,
   "version":"6.0.32 (a1c388)",
   "ipAddress":"172.27.24.26",
   "modelsUpdated":"2023-05-30 18:46:46",
   "modelPackageVersion":"6.0.23-1019~20230530170515~g6d5204",
   "bundleVersion":"60076",
'instance_info': {'id': 2, 'downCount': 8, 'upCount': 8, 'downTimeMs': 1685516996000000, 'downTime': '2023-05-31 07:09:56', 'version': '6.0.32 (a1c388)', 'ipAddress': '10.32.1.99'}

Vendor setup

A Darktrace user with Unrestricted Devices and Visualizer role.

Action

Steps

Obtain an API-Token Pair

Login to the Darktrace portal with your email and password.
Navigate to the System Config page on the Threat Visualizer of the instance you wish to request data from. Select “Settings” from the left-hand menu.
Locate the ‘API Token’ subsection and click ‘New’.
Two values will be displayed, a Public and Private token, the Private token will not be displayed again.

Both tokens are required to generate the DT-API-Signature value, which must be passed with every API request made to the appliance, so make sure you record them securely.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details
`base_url`	The Darktrace Respond API base URL (for example, https://euw1-1234-01.cloud.darktrace.com ).
`public_token`	The token obtained from Daktrace Respond for authentication.
`private_token`	The token obtained from Daktrace Respond for authentication.

Accepted authentication methods

Authentication method	Public token	Private token	Base URL
API-token pair	Required	Required	Required

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Events service

Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::CollectorDarktracePullerSetup(unknown,darktrace#darktrace_respond,summarystatistics#predefined) -> Starting the execution of setup()
INFO InputProcess::CollectorDarktracePullerSetup(unknown,darktrace#darktrace_respond,summarystatistics#predefined) -> Tokens have been validated successfully.
INFO InputProcess::CollectorDarktracePullerSetup(unknown,darktrace#darktrace_respond,summarystatistics#predefined) -> Finalizing the execution of setup()
INFO InputProcess::CollectorDarktracePullerSetup(unknown,darktrace#darktrace_respond,summarystatistics#predefined) -> Setup for module <CollectorDarktracePuller> has been successfully executed

Puller output

2023-03-27T16:22:12.290    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> The configuration property "start_time" is having a different value from previous collector executions so the persisted values will be removed. The new "start_time" value will be used as starting point, previous_value: "2023-03-10T00:00:00", new_value: "2023-03-25T00:00:00"
2023-03-27T16:22:12.291    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Starting data collection every 60 seconds
2023-03-27T16:22:12.291    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Starting a new pulling from darktrace at "2023-03-27T10:52:12.289241+00:00"
2023-03-27T16:22:12.297    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Total number of time slots to be processed: 58
2023-03-27T16:22:13.466    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 00:00:00+00:00 - 2023-03-25 01:00:00+00:00 is: 2
2023-03-27T16:22:14.476    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 01:00:00+00:00 - 2023-03-25 02:00:00+00:00 is: 0
2023-03-27T16:22:15.616    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 02:00:00+00:00 - 2023-03-25 03:00:00+00:00 is: 1
2023-03-27T16:22:16.640    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 03:00:00+00:00 - 2023-03-25 04:00:00+00:00 is: 0
2023-03-27T16:22:17.650    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 04:00:00+00:00 - 2023-03-25 05:00:00+00:00 is: 4
2023-03-27T16:22:18.669    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 05:00:00+00:00 - 2023-03-25 06:00:00+00:00 is: 1
2023-03-27T16:22:19.705    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 06:00:00+00:00 - 2023-03-25 07:00:00+00:00 is: 0
2023-03-27T16:22:21.122    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 07:00:00+00:00 - 2023-03-25 08:00:00+00:00 is: 1
2023-03-27T16:22:22.471    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 08:00:00+00:00 - 2023-03-25 09:00:00+00:00 is: 1
2023-03-27T16:22:23.902    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 09:00:00+00:00 - 2023-03-25 10:00:00+00:00 is: 0
2023-03-27T16:22:25.338    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 10:00:00+00:00 - 2023-03-25 11:00:00+00:00 is: 6
2023-03-27T16:22:26.571    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 11:00:00+00:00 - 2023-03-25 12:00:00+00:00 is: 0
2023-03-27T16:22:26.572    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of processed time slots so far: 12
2023-03-27T16:22:27.896    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Number of events received for time slot >> 2023-03-25 12:00:00+00:00 - 2023-03-25 13:00:00+00:00 is: 0

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2023-03-27T16:23:16.990    INFO InputProcess::CollectorDarktracePuller(darktrace,darktrace_respond,summarystatistics,predefined) -> Received response, messages(total/duplicated): 65/4/tag template used: "my.app.test2", avg_time_per_source_message: 6.022 ms

Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Error type	Error ID	Error Message	Cause	Solution
InitVariablesError	1	"{module_globals}" mandatory property is missing or empty	module_globals is not defined in the collector_definitions.yaml file	Define the module_globals in the collector_definitions file
InitVariablesError	2	"{module_globals}" mandatory property is missing or empty	module_globals field is not a dictionary	module_globals must be modified to be a dictionary
InitVariablesError	3	"base_url" mandatory property is missing or empty	base_url is missing from module_properties	Define base_url inside the module_properties
InitVariablesError	4	"base_url" property must be a string	base_url is not a string	base_url should be a string
InitVariablesError	5	"date_format" mandatory property is missing or empty	date_format is missing from module_properties	Define date_format inside the module_properties
InitVariablesError	6	"date_format" mandatory property is missing or empty	date_format is missing from module_properties	Define date_format inside the module_properties
InitVariablesError	7	"{module_properties_key_path}" mandatory property is missing or empty	module_properties is not defined in the collector_definitions.yaml file	Define the module_properties in the collector_definitions file
InitVariablesError	8	{module_properties_key_path}" property must be a dictionary	module_properties field is not a dictionary	module_properties must be modified to be a dictionary
InitVariablesError	9	"{module_properties_key_path}.endpoint" mandatory property is missing or empty	endpoint is not defined	deifne the endpoint
InitVariablesError	10	"{module_properties_key_path}.endpoint" property must be a string	endpoint is not a string	make endpoint a string
InitVariablesError	11	"{module_properties_key_path}.time_window_interval_in_minutes" mandatory property is missing or empty	time_window_interval_in_minutes is not present	define time_window_interval_in_minutes
InitVariablesError	12	"{module_properties_key_path}.time_window_interval_in_minutes" property must be an integer	time_window_interval_in_minutes is not an integer	time_window_interval_in_minutes must be an int
InitVariablesError	13	"{module_properties_key_path}.time_window_interval_in_minutes" property can not be a negative value	time_window_interval_in_minutes is less than 0	time_window_interval_in_minutes should be positive
InitVariablesError	14	"{module_properties_key_path}.devo_tag" mandatory property is missing or empty	devo_tag is not defined	define devo_tag
InitVariablesError	15	"{module_properties_key_path}.devo_tag" property must be a string	devo_tag is not a string	devo_tag must be a string
InitVariablesError	16	"{module_properties_key_path}.unique_id" property must be a string	unique_id is not a string	unique_id must be a string
InitVariablesError	17	"{module_properties_key_path}.persistence_version" mandatory property is missing or empty	persistence_version is not present	define persistence_version
InitVariablesError	18	"{module_properties_key_path}.persistence_version" property must be an integer	persistence_version is not an integer	persistence_version must be an int
InitVariablesError	19	"{module_properties_key_path}.persistence_version" property can not be a negative value	persistence_version is less than 0	persistence_version should be positive
InitVariablesError	20	inputs.darktrace mandatory property is missing or empty	inputs.darktrace is not present in config.yaml file	define inputs.darktrace in config.yaml file
InitVariablesError	21	inputs.darktrace property must be a dictionary	darktrace input is not a dict	make darktrace input a dict
InitVariablesError	22	inputs.darktrace.credentials mandatory property is missing or empty	credentials property is missing in config.yaml	credentials property must be defined
InitVariablesError	23	inputs.darktrace.credentials property must be a dictionary	credentials property is not a dict	credentials property must be a dict
InitVariablesError	24	inputs.darktrace.credentials.public_token mandatory property is missing or empty	public_token is missing in credentials section	public_token must be defined in the credentials section
InitVariablesError	25	inputs.darktrace.credentials.public_token property must be a string	public_token is not a string	public_token should be a string
InitVariablesError	26	inputs.darktrace.credentials.private_token mandatory property is missing or empt	private_token is missing in credentials section	private_token must be defined in the credentials section
InitVariablesError	27	inputs.darktrace.credentials.private_token property must be a string	private_token is not a string	private_token should be a string
InitVariablesError	28	inputs.darktrace.credentials.instance mandatory property is missing or empty	instance is not present in the credentials section	define the instance variable
InitVariablesError	29	inputs.darktrace.credentials.instance property must be a string	instance is not a string	instance has to be a string
InitVariablesError	30	inputs.{self.input_name}.services.{self.service_name} mandatory property is missing or empty	{self.service_name} is not defined inside services section	{self.service_name} has to be defined
InitVariablesError	31	inputs.{self.input_name}.services.{self.service_name} property must be a dictionary	{self.service_name} is not a dict	{self.service_name}must be a dict
InitVariablesError	32	inputs.{self.input_name}.services.{self.service_name}.override_devo_tag property must be a string	override_devo_tag is not a string	override_devo_tag must be a string
InitVariablesError	33	inputs.{self.input_name}.services.{self.service_name}.start_time_in_utc_format property must be a string	start_time_in_utc_format is not a string	start_time_in_utc_format must be a string
InitVariablesError	34	inputs.{self.input_name}.services.{self.service_name}.start_time_in_utc_format does not match the format {self.collector_variables["date_format"]}	start_time_in_utc_format is not proper	start_time_in_utc_format must be consistent with {self.collector_variables["date_format"]}
InitVariablesError	35	inputs.{self.input_name}.services.{self.service_name}.override_time_window_interval_in_minutes property must be an int	override_time_window_interval_in_minutes property is not an integer	override_time_window_interval_in_minutes property must be an integer
PullError	300	HTTP Error occurred while retrieving events from Darktrace server	Some unknown exception happened while making the HTTP request	Reach out to the developer with the exact error message
PullError	301	Some error occurred while retrieving events from Darktrace server.	An error occured apart from HTTP while making request to the server	Reach out to the developer with the exact error message

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Verify collector operations

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

2023-03-27T16:22:07.976    INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-1-local.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-03-27T16:22:07.976    INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-03-27T16:22:07.976    INFO MainProcess::MainThread -> "/etc/devo/job" does not exists
2023-03-27T16:22:07.977    INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-03-27T16:22:07.977    INFO MainProcess::MainThread -> "/etc/devo/collector" does not exists
2023-03-27T16:22:07.977    INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "/Users/krishan.dhingra/devo/github/devo-collector-darktrace/config/config-1-local.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
2023-03-27T16:22:08.080    INFO MainProcess::MainThread -> Build time: "UNKNOWN", OS: "macOS-12.6.1-x86_64-i386-64bit", collector(name:version): "darktrace_collector:1.0.0", owner: "aaa.bbb@domain.com", started at: "2023-03-27T10:52:08.059847Z"
2023-03-27T16:22:08.092    INFO MainProcess::MainThread -> Initialized all object from "MainProcess" process

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

2023-03-27T16:22:09.310    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-03-27T16:22:09.311    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_eu_1) -> Starting thread (every 300 seconds)
2023-03-27T16:22:09.311    INFO InputProcess::MainThread -> [GC] global: 66.6% -> 66.6%, process: RSS(44.88MiB -> 44.88MiB), VMS(32.80GiB -> 32.80GiB)
2023-03-27T16:22:09.312    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_eu_1) -> Starting thread
2023-03-27T16:22:09.312    INFO InputProcess::MainThread -> global_status: {"input_process": {"process_id": 48693, "process_status": "running", "thread_counter": 11, "thread_names": ["MainThread", "pydevd.Writer", "pydevd.Reader", "pydevd.CommandThread", "pydevd.CheckAliveThread", "QueueFeederThread", "CollectorDarktracePullerSetup(unknown,darktrace#darktrace_respond,aianalyst/incidentevents#predefined)", "CollectorDarktracePuller(darktrace,darktrace_respond,aianalyst/incidentevents,predefined)", "QueueFeederThread", "ServiceThread(darktrace,darktrace_respond,aianalyst/incidentevents,predefined)", "InputThread(darktrace,darktrace_respond)"], "memory_info": {"rss": "44.90MiB", "vms": "32.80GiB", "pfaults": "12.52KiB", "pageins": "1.00B"}, "input_threads": [[]], "running_flag": true, "message_queues": {"standard": {"name": "standard_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 1, "put_lock": "<Lock(owner=unknown)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x10445bb50>"}, "lookup": {"name": "lookup_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 0, "put_lock": "<Lock(owner=unknown)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x104468190>"}, "internal": {"name": "internal_queue_multiprocessing", "max_size_in_messages": 10000, "max_size_in_mb": 1024, "max_wrap_size_in_items": 100, "current_size": 10, "put_lock": "<Lock(owner=unknown)>", "input_lock": "<multiprocessing.synchronize.Event object at 0x10446e8b0>"}}}}
2023-03-27T16:22:09.313    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services	Description
`internal_senders`	In charge of delivering internal metrics to Devo such as logging traces or metrics.
`standard_senders`	In charge of delivering pulled events to Devo.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Standard - Total number of messages sent: 57, messages sent since "2023-01-10 16:09:16.116750+00:00": 0 (elapsed 0.000 seconds

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2023-01-10 16:09:16.116750+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.

By default these traces will be shown every 10 minutes.

Check memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

2023-03-27T16:22:09.311    INFO InputProcess::MainThread -> [GC] global: 66.6% -> 66.6%, process: RSS(44.88MiB -> 44.88MiB), VMS(32.80GiB -> 32.80GiB)
2023-03-27T16:22:09.350    INFO OutputProcess::MainThread -> [GC] global: 66.6% -> 66.6%, process: RSS(45.21MiB -> 45.34MiB), VMS(32.89GiB -> 32.89GiB)

Change log

Release	Released on	Release type	Details	Recommendations
`v1.0.0`	11 May 2023	FIRST RELEASE	Released the first version of the Darktrace Respond collector.	`Recommended version`