Microsoft Defender ATP for Endpoint collector

[ 1 Overview ] [ 2 Configuration requirements ] [ 3 Devo collector features ] [ 4 Data sources ] [ 5 Flattening preprocessing ] [ 6 Vendor setup ] [ 7 Minimum configuration required for basic pulling ] [ 8 Accepted authentication methods ] [ 9 Run the collector ] [ 10 Collector services detail ] [ 11 Collector operations ] [ 12 Change log ]

Overview

Microsoft Defender for Endpoint, formerly Microsoft Defender Advanced Threat Protection (Defender ATP), provides enterprise-level protection to endpoints to prevent, detect, investigate, and respond to advanced threats.

The Devo Defender for Endpoint Collector enables you to retrieve data from the listed sources below via Microsoft Defender for Endpoints APIs into Devo query, correlate, analyze, and visualize to enable Enterprise IT and Cybersecurity teams to take decisions at the petabyte scale.

Configuration requirements

To run this collector, there are some configurations detailed below that you need to take into account.

Configuration	Details

Configuration	Details
Microsoft Defender account	You need to have a Microsoft Defender account. Refer to Vendor setup section to see how to do it.
Microsoft Azure account	You need to have a Microsoft Azure account. Refer to Vendor setup section to see how to do it.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`Not allowed`
Running environments	`Collector server` `On-premise`
Populated Devo events	`Table`
Flattening preprocessing	`Yes (optional)`

Data sources

Data source	Description	API endpoint	Collector service name	Devo table	Available from release

Data source	Description	API endpoint	Collector service name	Devo table	Available from release
Alerts	Alerts data source shows a list of alerts that were flagged from devices in your network. They can be filtered by these topics: domain file ip machine user	List alerts: `https://api.securitycenter.microsoft.com/api/alerts` Alert details: `https://api.securitycenter.microsoft.com/api/alerts/{{alert_id}}` Alerts related to other `{{topic}}:` `https://api.securitycenter.microsoft.com/api/alerts/{{alert_id}}/{{topic}}`	`alerts`	`edr.microsoft_defender.endpoint.alerts` Events The events ingested by this table are Microsoft Defender Identity Events.	`v1.0.0`
Vulnerabilities	Vulnerabilities data source offers continuous vulnerability discovery and assessment. It can be filtered by this topic: machine	List vulnerabilities: `https://api.securitycenter.microsoft.com/api/vulnerabilities` Vulnerability details: `https://api.securitycenter.microsoft.com/api/vulnerabilities/{{vulnerability_id}}` Alerts related to machine: `https://api.securitycenter.microsoft.com/api/vulnerabilities/{{vulnerability_id}}/machineReferences`	`vulnerabiities`	`edr.microsoft_defender.endpoint.vulnerabilities`	`v1.0.0`
Machines	Machines data source gets a list of devices in your network. They can be filtered by these topics: user alert software vulnerability recommendation This is a snapshot-like service able to run in periods no shorter than 1h.	List machines: `https://api.securitycenter.microsoft.com/api/machines` Machine details: `https://api.securitycenter.microsoft.com/api/machines/{{machine_id}}` Machines related to other `{{topic}}`: `https://api.securitycenter.microsoft.com/api/machines/{{machine_id}}/{{topic}}`	`machines`	`edr.microsoft_defender.endpoint.machines`	`v1.0.0`
Software	Software data source gets a list of software installed in devices in your network. It can be filtered by these topics: Vulnerability Machine Distribution Missing KB.	List software: `https://api.securitycenter.microsoft.com/api/software` Software details: `https://api.securitycenter.microsoft.com/api/software/{{software_id}}` Software related to other `{{topic}}`: `https://api.securitycenter.microsoft.com/api/software/{{software_id}}/{{topic}}`	`softwares`	`edr.microsoft_defender.endpoint.software`	`v1.0.0`
Recommendations	Recommendations data source lists security recommendations for devices in your network. It can be filtered by these topics: Software Machine Vulnerability	List recommendations: `https://api.securitycenter.microsoft.com/api/recommendations` Recommendation details: `https://api.securitycenter.microsoft.com/api/recommendations/{{recommendation_id}}` Recommendations related to other `{{topic}}`: `https://api.securitycenter.microsoft.com/api/recommendations/{{recommendation_id}}/{{topic}}`	`recommendations`	`edr.microsoft_defender.endpoint.recommendations`	`v1.0.0`
Investigations	Investigations data source lists all the automated investigations done by MS Defender in your network.	`https://api.securitycenter.microsoft.com/api/investigations`	`investigations`	`edr.microsoft_defender.endpoint.investigations`	`v1.0.0`
Advanced Hunting	Advance Hunting data source gives the possibility to run custom queries.	`https://api.securitycenter.microsoft.com/api/advancedqueries/run`	`advanced_huntuing`	custom table: `my.app.xxx.yyy`	`v1.0.0`

For more information on how the events are parsed, visit our page.

Flattening preprocessing

In order to improve the data exploitation and enrichment, this collector is able to apply some flattering actions to the collected data before delivering it to Devo.

What is Flattening?

Flattening is used when some data is nested into the data structure, it is used to be faster on data exploitation, this process re-shapes the data structure to do so. There are different ways to flatten data, but the most used are applied over objects or arrays.

Flattening over objects

When flattening over objects, it creates new keys whose names are the combination of the external/internal keys.

Original structure	Result

Original structure	Result
`{ 'machine': { 'hostname': 'machine1', 'os': 'linux', 'ip': '1.2.3.4' } }`	`{ 'machine_hostname': 'machine1', 'machine_os': 'linux', 'machine_ip': '1.2.3.4' }`

Flattening over arrays

When flattening over arrays it replicates the event using one element of the array in each replicated event:

Example

Take into account that this flattening method generates more events than those originally collected. In the example below 1 single event, 3 events are generated. In the case of an event with multiple fields that contains an array, the number of events generated per original event is obtained by multiplying the number of elements in each array.

For instance, given an event with 3 fields that contains arrays with 2, 3 and 4 elements each, if we flatten this event, we’re going to generate 24 events from this single event (2*3*4 = 24).

Original structure	Result

Original structure	Result
`{ 'machine_hostname': 'machine1', 'machine_os': 'linux', 'ips':['1.2.3.4', '4.3.2.1', '1.1.1.1'] } # Flattened data structure (3 events generated from the original one)`

Enrichment techniques

The enrichment made in this collector consists of adding some fields to the original message, where the count of affected elements is displayed. This field uses the prefix_related.

The enrichment process is optional. In the configuration, it can be defined which entities must be enriched, for each data source. Refer to the Service Detail section.

The flattening cases processed in this collector are:

Data Source	Collector Service	Type	Behavior Details

Data Source

Collector Service

Type

Behavior Details

Alerts

alerts

Flattening over objects

When relatedUser is received as an alert detail, the flattening is applied as shown:

Received data (an object):

Flattened data:

Flattening over arrays

When evidence is received as an alert detail, the flattening is applied as shown:

Received data (an object):

Flattened and enriched data:

Alerts
Machines
Software
Vulnerabilities
Recommendations

alerts
machines
software
vulnerabilities
recommendations

Enrichment

Visit the Service Details section for details.

Vendor setup

There are some minimal requirements to enable this collector:

Microsoft Azure account.
Microsoft Defender account.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

Setting	Details

Setting	Details
`client_id`	Application (client) ID of the application created during the setup.
`client_secret`	Client secret created during the setup.
`tenant_id`	Tenant ID of the application created during the setup.
`override_api_base_url`	[optional] The URL of the REST API server for your MicrosoftDefender tenant. The default is : https://api.securitycenter.microsoft.com
`override_token_url`	[optional] The URL to get a Bearer token . The default is : https://login.microsoftonline.com

Accepted authentication methods

Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials configuration block.

Authentication Method	Client ID	Client Secret	Tenant ID

Authentication Method	Client ID	Client Secret	Tenant ID
Client ID / Client Secret / Tenant ID	REQUIRED	REQUIRED	REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Events service

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_poll_datetime parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

This service uses the field lastUpdateTime as a time reference. The goal is to detect the evolution an alert has had. The alerts detected in Microsoft Defender can be updated. For example, they can update the status from New to In Progress or to Resolved.

This means the API can return the same alert ID multiple times, and it can be read and sent to Devo at different moments. However, keep in mind that these are not duplicate events, but the same event with a modification applied.

Devo categorization and destination

All events are ingested into table edr.microsoft_defender.endpoint.alerts.

Data enrichment

Entity	Config paratemeter	Description	Endpoint	Result

Entity	Config paratemeter	Description	Endpoint	Result
files	`request_alert_related_files`	Enrich the alert with the counter of related files.	`/api/alerts/{alert_id}/files`	Add to the final message the field: `{..., 'related_files': int}`
ips	`request_alert_related_ips`	Enrich the alert with the counter of related ips.	`/api/alerts/{alert_id}/ips`	Add to the final message the field: `{..., 'related_ips': int}`
machines	`request_alert_related_machines`	Enrich the alert with the counter of related machines.	`/api/alerts/{alert_id}/machine`	Add to the final message the field: `{..., 'related_machines': int}`
domains	`request_alert_related_domains`	Enrich the alert with the counter of related domains.	`/api/alerts/{alert_id}/domains`	Add to the final message the field: `{..., 'related_domains': int}`
users	`request_alert_related_users`	Enrich the alert with the counter of related users.	`/api/alerts/{alert_id}/user`	Add to the final message the field: `{..., 'related_users': int}`

Verify data collection

Puller Output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

All events of this service are ingested into table edr.microsoft_defender.endpoint.machines.

Data Enrichment

Entity	Config paratemeter	Description	Endpoint	Result

Entity	Config paratemeter	Description	Endpoint	Result
logon_users	`request_machine_logon_users`	Enrich the machine with the counter of related logged on users.	`/api/machines/{machine_id}/logonusers`	Add to the final message the field: `{..., 'related_logon_users': int}`
alerts	`request_machine_related_alerts`	Enrich the machine with the counter of related alerts.	`/api/machines/{machine_id}/alerts`	Add to the final message the field: `{..., 'related_alerts': int}`
vulnerabilities	`request_machine_vulnerabilities`	Enrich the machine with the counter of related vulnerabilities.	`/api/machines/{machine_id}/vulnerabilities`	Add to the final message the field: `{..., 'related_vulnerabilities': int}`
recommendations	`request_machine_security_recommendations`	Enrich the machine with the counter of related recommendations.	`/api/machines/{machine_id}/recommendations`	Add to the final message the field: `{..., 'related_recommendations': int}`

Verify data collection

Puller Output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

All events of this service are ingested into table edr.microsoft_defender.endpoint.software.

Data Enrichment

Entity	Config paratemeter	Description	Endpoint	Result

Entity	Config paratemeter	Description	Endpoint	Result
vulnerabilities	`request_vulnerabilities_by_software`	Enrich the software with the counter of related vulnerabilities.	`/api/software/{soft_id}/vulnerabilities`	Add to the final message the field: `{..., 'related_vulnerabilities': int}`
missing kbs	`request_softtware_missing_kbs`	Enrich the software with the counter of related missing KBs.	`/api/software/{soft_id}/getmissingkbs`	Add to the final message the field: `{..., 'related_missing_kbs': int}`
machines	`request_machines_by_software`	Enrich the software with the counter of related machines.	`/api/software/{soft_id}/machineReferences`	Add to the final message the field: `{..., 'related_machines': int}`
distributions	`request_softtware_version_distributions`	Enrich the software with the counter of related version distributions.	`/api/software/{soft_id}/distributions`	Add to the final message the field: `{..., 'related_version_distribution': int}`

Verify data collection

Puller Output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

All events of this service are ingested into table edr.microsoft_defender.endpoint.vulnerabilities

Data Enrichment

Entity	Config paratemeter	Description	Endpoint	Result

Entity

Config paratemeter

Description

Endpoint

Result

machines

request_machine_by_vulnerabilities

Enrich the vulnerability with the counter of related machines.

/api/vulnerabilities/{vuln_id}/machineReferences

Add to the final message the field:

{..., 'related_machines': int}

Verify data collection

Puller Output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Change log

Release	Released on	Release type	Details	Recommendations

Release	Released on	Release type	Details	Recommendations
`v1.4.0`	Sep 18, 2024	IMPROVEMENT	Improvements Updated `DevoCollectorSDK` from `v1.11.1` to `v1.12.4` Added new sender for relay in house + TLS Added persistence functionality for gzip sending buffer Added Automatic activation of gzip sending Improved behaviour when persistence fails Upgraded DevoSDK dependency Fixed console log encoding Restructured python classes Improved behaviour with non-utf8 characters Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB) New persistence format/structure (compression in some cases) Removed dmesg execution (It was invalid for docker execution) Applied changes to make DCSDK compatible with MacOS Upgrade DevoSDK dependency to version v5.4.0 Change internal queue management for protecting against OOMK Extracted ModuleThread structure from PullerAbstract Improve Controlled stop when both processes fails to instantiate Improve Controlled stop when InputProcess is killed Bug related to lost of collector_name , collector_id and job_id Bug retaled queues and ValueError Updated the docker base image to 1.3.0	`Recommended version`
`v1.3.0`	Jun 17, 2024	IMPROVEMENT	Improvements Updated `DevoCollectorSDK` to `v1.11.1'	`Update`
`v1.2.0`	Mar 4, 2024	IMPROVEMENT BUG FIXING	Improvements Updated Docker image base to 1.2.0 Updated `DevoCollectorSDK` to `v1.11.0' Fix for SyslogSender related to UTF-8 Enhance of troubleshooting. Trace Standardization, Some traces has been introduced. Introduced a mechanism to detect "Out of Memory killer" situation Bug fixing Changed the persistence logic related to 'lastPollTime' to solve the bug related to duplication of events	`Update`
`v1.1.1`	Dec 4, 2023	FEATURE BUG FIX	New features Devo Collector SDK upgraded from `1.8.0` to `1.10.2`. Bug fixing Fix a use case that allowed sending duplicate events to Devo for the Alerts service, as the last IDs sent were deleted before required. Fix a use case that avoided sending some events to Devo for the Alerts service, as the time of the latest sent event wasn’t appropriately persisted.	`Update`
`v1.1.0`	Jun 7, 2023	FEATURE BUG FIX	New features Machines service adapted to work as a snapshot-like service, instead of incrementally. Devo Collector SDK upgraded from 1.4.4 to 1.8.0. Bug fixing Upgrading the Devo Collector SDK solved a problem related to connection loss.	`Update`
`v1.0.1`	Jan 17, 2023	FEATURE BUG FIX	New features Confirmed that only Application permissions are enough to run the collector. Devo Collector SDK upgraded from 1.4.2 to 1.4.4. Bug fixing Content Type header for Advanced Hunting set to `application/json`	`Upgrade`
`v1.0.0`	Sep 20, 2022	FEATURE	New features Alerts created in MS Defender are collected and enriched. The enrichment includes information related to evidences, domains, files, IPs, devices and users. Machine information enriched with information related to logged on users, alerts, installed software, vulnerabilities and security recommendations. Snapshot of software installed and enriched with data related to version distributions, machine, vulnerabilities and missing software updates. Vulnerabilities information enriched with machines affected by the vulnerability. Recommendations information snapshot, including information related to specific software, machines and vulnerabilities. Investigations performed. Advanced hunting, which allows making custom queries in Kusto Query Language.	`-`