To run this collector, there are some configurations detailed below that you need to take into account.
Configuration
Details
Configuration
Details
Microsoft Defender account
You need to have a Microsoft Defender account. Refer to Vendor setup section to see how to do it.
Microsoft Azure account
You need to have a Microsoft Azure account. Refer to Vendor setup section to see how to do it.
Overview
Microsoft Defender for Endpoint, formerly Microsoft Defender Advanced Threat Protection (Defender ATP), provides enterprise-level protection to endpoints to prevent, detect, investigate, and respond to advanced threats.
The Devo Defender for Endpoint Collector enables you to retrieve data from the listed sources below via Microsoft Defender for Endpoints APIs into Devo query, correlate, analyze, and visualize to enable Enterprise IT and Cybersecurity teams to take decisions at the petabyte scale.
Devo collector features
Feature
Details
Feature
Details
Allow parallel downloading (multipod)
Not allowed
Running environments
Collector server
On-premise
Populated Devo events
Table
Flattening preprocessing
Yes (optional)
Data sources
Data source
Description
API endpoint
Collector service name
Devo table
Available from release
Data source
Description
API endpoint
Collector service name
Devo table
Available from release
Alerts
Alerts data source shows a list of alerts that were flagged from devices in your network. They can be filtered by these topics:
For more information on how the events are parsed, visit our page.
Flattening preprocessing
In order to improve the data exploitation and enrichment, this collector is able to apply some flattering actions to the collected data before delivering it to Devo.
What is Flattening?
Flattening is used when some data is nested into the data structure, it is used to be faster on data exploitation, this process re-shapes the data structure to do so. There are different ways to flatten data, but the most used are applied over objects or arrays.
Flattening over objects
When flattening over objects, it creates new keys whose names are the combination of the external/internal keys.
When flattening over arrays it replicates the event using one element of the array in each replicated event:
Example
Take into account that this flattening method generates more events than those originally collected. In the example below 1 single event, 3 events are generated. In the case of an event with multiple fields that contains an array, the number of events generated per original event is obtained by multiplying the number of elements in each array.
For instance, given an event with 3 fields that contains arrays with 2, 3 and 4 elements each, if we flatten this event, we’re going to generate 24 events from this single event (2*3*4 = 24).
Original structure
Result
Original structure
Result
{
'machine_hostname': 'machine1',
'machine_os': 'linux',
'ips':['1.2.3.4', '4.3.2.1', '1.1.1.1']
}
# Flattened data structure (3 events generated from the original one)
Enrichment techniques
The enrichment made in this collector consists of adding some fields to the original message, where the count of affected elements is displayed. This field uses the prefix_related.
The enrichment process is optional. In the configuration, it can be defined which entities must be enriched, for each data source. Refer to the Service Detail section.
The flattening cases processed in this collector are:
Data Source
Collector Service
Type
Behavior Details
Data Source
Collector Service
Type
Behavior Details
Alerts
alerts
Flattening over objects
When relatedUser is received as an alert detail, the flattening is applied as shown:
Received data (an object):
Flattened data:
Flattening over arrays
When evidence is received as an alert detail, the flattening is applied as shown:
Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials configuration block.
Authentication Method
Client ID
Client Secret
Tenant ID
Authentication Method
Client ID
Client Secret
Tenant ID
Client ID / Client Secret / Tenant ID
REQUIRED
REQUIRED
REQUIRED
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Events service
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the historical_poll_datetime parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
All events are ingested into table edr.microsoft_defender.endpoint.alerts.
Data Enrichment
Entity
Config paratemeter
Description
Endpoint
Result
Entity
Config paratemeter
Description
Endpoint
Result
files
request_alert_related_files
Enrich the alert with the counter of related files.
/api/alerts/{alert_id}/files
Add to the final message the field:
{..., 'related_files': int}
ips
request_alert_related_ips
Enrich the alert with the counter of related ips.
/api/alerts/{alert_id}/ips
Add to the final message the field:
{..., 'related_ips': int}
machines
request_alert_related_machines
Enrich the alert with the counter of related machines.
/api/alerts/{alert_id}/machine
Add to the final message the field:
{..., 'related_machines': int}
domains
request_alert_related_domains
Enrich the alert with the counter of related domains.
/api/alerts/{alert_id}/domains
Add to the final message the field:
{..., 'related_domains': int}
users
request_alert_related_users
Enrich the alert with the counter of related users.
/api/alerts/{alert_id}/user
Add to the final message the field:
{..., 'related_users': int}
Verify data collection
Puller Output
A successful initial run has the following output messages for the puller module:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
All events of this service are ingested into table edr.microsoft_defender.endpoint.machines.
Data Enrichment
Entity
Config paratemeter
Description
Endpoint
Result
Entity
Config paratemeter
Description
Endpoint
Result
logon_users
request_machine_logon_users
Enrich the machine with the counter of related logged on users.
/api/machines/{machine_id}/logonusers
Add to the final message the field:
{..., 'related_logon_users': int}
alerts
request_machine_related_alerts
Enrich the machine with the counter of related alerts.
/api/machines/{machine_id}/alerts
Add to the final message the field:
{..., 'related_alerts': int}
vulnerabilities
request_machine_vulnerabilities
Enrich the machine with the counter of related vulnerabilities.
/api/machines/{machine_id}/vulnerabilities
Add to the final message the field:
{..., 'related_vulnerabilities': int}
recommendations
request_machine_security_recommendations
Enrich the machine with the counter of related recommendations.
/api/machines/{machine_id}/recommendations
Add to the final message the field:
{..., 'related_recommendations': int}
Verify data collection
Puller Output
A successful initial run has the following output messages for the puller module:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
All events of this service are ingested into table edr.microsoft_defender.endpoint.software.
Data Enrichment
Entity
Config paratemeter
Description
Endpoint
Result
Entity
Config paratemeter
Description
Endpoint
Result
vulnerabilities
request_vulnerabilities_by_software
Enrich the software with the counter of related vulnerabilities.
/api/software/{soft_id}/vulnerabilities
Add to the final message the field:
{..., 'related_vulnerabilities': int}
missing kbs
request_softtware_missing_kbs
Enrich the software with the counter of related missing KBs.
/api/software/{soft_id}/getmissingkbs
Add to the final message the field:
{..., 'related_missing_kbs': int}
machines
request_machines_by_software
Enrich the software with the counter of related machines.
/api/software/{soft_id}/machineReferences
Add to the final message the field:
{..., 'related_machines': int}
distributions
request_softtware_version_distributions
Enrich the software with the counter of related version distributions.
/api/software/{soft_id}/distributions
Add to the final message the field:
{..., 'related_version_distribution': int}
Verify data collection
Puller Output
A successful initial run has the following output messages for the puller module:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
All events of this service are ingested into table edr.microsoft_defender.endpoint.vulnerabilities
Data Enrichment
Entity
Config paratemeter
Description
Endpoint
Result
Entity
Config paratemeter
Description
Endpoint
Result
machines
request_machine_by_vulnerabilities
Enrich the vulnerability with the counter of related machines.
/api/vulnerabilities/{vuln_id}/machineReferences
Add to the final message the field:
{..., 'related_machines': int}
Verify data collection
Puller Output
A successful initial run has the following output messages for the puller module:
After a successful collector’s execution (this is, no error logs were found), you should be able to see the following log message:
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Change log for v1.x.x
Release
Released on
Release type
Details
Recommendations
Release
Released on
Release type
Details
Recommendations
v1.0.0
Sep 20, 2022
FEATURE
New features:
Alerts created in MS Defender are collected and enriched. The enrichment includes information related to evidences, domains, files, IPs, devices and users.
Machine information enriched with information related to logged on users, alerts, installed software, vulnerabilities and security recommendations.
Snapshot of software installed and enriched with data related to version distributions, machine, vulnerabilities and missing software updates.
Vulnerabilities information enriched with machines affected by the vulnerability.
Recommendations information snapshot, including information related to specific software, machines and vulnerabilities.
Investigations performed.
Advanced hunting, which allows making custom queries in Kusto Query Language.
