Microsoft Graph collector
- Juan TomƔs Alonso Nieto (Deactivated)
- Ćngel GarcĆa
- Former user (Deleted)
If you are migrating from any 1.x version, please read this section.
Overview
The Microsoft Graph Collector provides the ability to collect data and intelligence from services such as Microsoft 365, Windows, and Enterprise Mobility and Security. Currently, this data collector is able to ingest only security alerts, scores, provisioning, audit and sign ins retrieved from the Microsoft products. This empowers customers to streamline security operations and better defend against increasing cyber threats faced in their Azure AD and Microsoft 365 environments and beyond.
Devoās Microsoft Graph collector also enables customers to correlate events and context to improve threat protection and response, and includes key entities described in the next sections.
Configuration requirements
To run this collector, there are some configurations detailed below that you need to consider.
Configuration | Details |
|---|---|
Azure account | Azure account with admin level permissions and Azure AD tenant. |
Credentials | The credentials configuration block has been filled correctly. |
More information
Refer to the Vendor setup section to know more about these configurations.
Devo collector features
Feature | Details |
|---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Allowed source events obfuscation |
|
Data sources
Data source | Description | API endpoint | Collector service name | Devo table |
|---|---|---|---|---|
Audit logs - provisioning | Represents an action performed by the Microsoft Entra provisioning service and its associated properties. |
|
|
|
Audit logs - directory | Represents the directory audit items and its collection. |
|
|
|
Audit logs - sign-ins | Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API. |
|
|
|
Audit logs - sign-ins (v2) | Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API. |
|
|
|
Legacy Alerts | This resource corresponds to the first generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft or a partner security solution has identified. This type of alerts federates calling of supported Azure and Microsoft 365 Defender security providers listed in Use the Microsoft Graph security API. It aggregates common alert data among the different domains to allow applications to unify and streamline management of security issues across all integrated solutions. |
|
|
|
Alerts (v2) | This resource corresponds to the latest generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft 365 Defender, or a security provider integrated with Microsoft 365 Defender, has identified. When detecting a threat, a security provider creates an alert in the system. Microsoft 365 Defender pulls this alert data from the security provider, and consumes the alert data to return valuable clues in an alert resource about any related attack, impacted assets, and associated evidence. It automatically correlates other alerts with the same attack techniques or the same attacker into an incident to provide a broader context of an attack. Aggregating alerts in this manner makes it easy for analysts to collectively investigate and respond to threats. |
|
|
|
Secure Scores | Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held. This data is sorted by createdDateTime, from latest to earliest. This will allow you to page responses by using $top=n, where n = the number of days of data that you want to retrieve. |
|
|
|
Secure Scores Control Profiles | Represents a tenant's secure score per control data. By default, this resource returns all controls for a tenant and can explicitly pull individual controls. |
|
|
|
API limits, delays and known issues
The Graph API imposes some throttling limits to the applications. The number of requests in a given time is limited according to several criteria. There is a generic limitation for the Graph API and service specific limits.
Any request can be evaluated against multiple limits, depending on the scope of the limit (per app across all tenants, per tenant for all apps, per app per tenant, and so on), the request type (GET, POST, PATCH, and so on), and other factors. The first limit to be reached triggers throttling behavior.
In the case of the services used by the collector, the limits are:
Request type | Per app across all tenants |
|---|---|
Any | 130,000 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 150 requests per minute |
| 150 requests per minute |
| 150 requests per minute or 10,000 in a 10-minute period |
| 10,000 in a 10-minute period |
A complete reference of the limits can be found here.
When the collector reaches the API limit, the API returns a 429 error code. This is reflected in the logs of the collector with a message like 429 Error during API call. If several of these messages are found, the solution is to use the request_period_in_seconds for this service to decrease the frequency of calls.
For instance, a request period of 300 means that the API will be called every 5 minutes for this service, instead of the default value of 60 seconds.
API delays
The Graph API has some peculiarities that we need to consider to get optimal results. Events are only sometimes available in the Graph API in real time or near real time. Most events arrive seconds after they actually occur, but some can arrive minutes or even hours late. Microsoft SLAs can be anywhere from 3 minutes to 6 hours in most cases. Check more information here.
So, trying to fetch the events in near real time sometimes causes the collector to miss events. For instance, if we try to get the events at 12:00 and we ask the API for events in the range 11:55ā12:00, we will get the existing events with event dates between 11:55 and 12:00, as they are stored in the Graph API at 12:00. But a new event with an internal event date of 11:58 may arrive at 12:03, for example. In this case, the collector would not gather the new event when asking the API for events from 12:00 to 12:05, because the Graph API considers the event date of 11:58 to be out of this range.
The solution to this issue is to ask the API for events after some time, not immediately. For instance, we can ask for events in the range of 11:55ā12:00 at 13:00 instead of trying to get them right at 12:00. This way, if an event arrives at 12:03, the Graph API will include it in the events gathered for the range.
An optional parameter allows us to postpone data gathering. The override_fetch_gap_seconds parameter value is the number of seconds we want to wait fordata gathering. For instance, if we set it to 3600 seconds, the collector gathers the events an hour late; it asks for events from 11:55ā12:00 at 13:00, and so on.
There are two caveats to consider:
Using this parameter delays the appearance of events in Devo. For some applications, this delay could be unacceptable; some applications require near real time operation.
Sometimes, when we start the collector, no events are received for a while. For instance, using a delay of 3600 seconds, if we start the collector at 12:00 with a
start_time_in_utcof 12:00, the first events will not arrive in Devo until 13:00.
Vendor setup
Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:
An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.
You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.
You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits.
Ā | Action | Steps |
|---|---|---|
1 | Register and configure the application |
|
2 | Grant the required permissions |
|
3 | Obtain the requires credentials for the collector |
|
Permission reference per service
Collector service | Resource | Required permissions | Microsoft documentation |
|---|---|---|---|
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
Required for all services |
|
|
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
Setting | Details |
|---|---|
| This is the Tenantās ID you created in Azure AD. You can obtain it from the Overview page in your registered application. |
| This is the Clientās ID you created in Azure AD. You can obtain it from the Overview page in your registered application. |
| This is the Clientās secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. |
Accepted authentication methods
This collector only accepts one single authentication method. You will have to fill the following properties on the credentials configuration block:
Authentication method | Tenant ID | Client ID | Client secret |
|---|---|---|---|
REQUIRED | REQUIRED | REQUIRED |
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Restart the persistence for a service
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utcparameter to a different one.Save the changes.
Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Common logic
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
| 1 | Invalid | The configured | Update the |
| 2 | Invalid | The configured | Update the |
| 101 | Failed to fetch OAuth token from | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
| 102 | Failed to fetch data from | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
| 401 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 401 response. | Ensure that the provided credentials are correct and provide read access to the targeted data. |
| 429 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 429 response. | The collector will attempt to retry requests (default up to 3 times) and |
| 498 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 500 response. | If the API returns a 500 but successfully completes subsequent runs then you may ignore this error. If the API repeatedly returns a 500 error, ensure the server is reachable and operational. |
Collector operations
Verify collector operations
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB ->
410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB ->
705.28MiB)Change log1.X to 2.X migration guide
Version 2.0.0 introduced several changes to the collector's configuration. When upgrading the collector, users must make changes to their configuration file to ensure that the collector continues to work as expected.
In versions 1.X of the collector, some services had a config parameter tag_version with values v1or v2. The effect of this parameter is that the destination table for these services will be different. These are the destination tables according to the tag_version value:
v1 | v2 (default) |
|---|---|
|
|
|
|
In the new collector 2.0.0, the old config parameter tag_version has been removed. The same effect as v1can be made using override_tag, with these values:
Service |
|
|---|---|
|
|
|
|
For instance, if we have this in the old config:
you should change it to:
In old versions of this collector, all alerts were sent to the table cloud.msgraph.security.alerts
However, as there are alerts of several types, in version 2.0.0 alerts are categorized according to their vendorInformation-provider field (their vendor) and sent to different tables:
Vendor | Old table | New table |
|---|---|---|
IPC |
|
|
MCAS |
| |
Microsoft Defender ATP |
| |
Microsoft 365 Defender |
| |
Office 365 Security and Compliance |
| |
Azure Sentinel |
| |
ASC |
| |
Azure Advanced Threat Protection |
| |
Others |
|
It is possible to avoid this feature and send all alerts to the same table by editing the config file and changing the tag:
Config file comparison
Below you will find a sample 1.7.1 configuration file and a sample 2.0.0 configuration file.
Config file changes
The URL endpoints (
override_base_url_main,override_base_url_vendor,override_base_url_vendor_with_sub_provider,override_login_url) have been moved from the individual services to the global configuration section.override_base_url_mainhas been renamed tooverride_base_url.tag_versionhas been removed.pull_sliding_window_timespan_periodhas been removed.reset_persistence_authhas been removed.override_time_delta_in_dayshas been removed.ms_365_environmenthas been replaced byoverride_top_level_domain. GCC High Gov should useusin theoverride_top_level_domain. Alternatively, users can useoverride_base_urlto specify the GCC High Gov base URL.additional_filterhas been added to all services. Users can use this field to specify additional filters that will be applied when querying the Microsoft Graph API.The collector can use new services from Graph (beta endpoint in Graph), that services, that use to be in a separate service for each type, have been consolidated into one service called
signIns_v2. Users should remove all three services from their config and use only thesignIns_v2service.start_timehas been renamed tostart_time_in_utc.
Persistence changes
The persistence object consists of the following fields:
persistence_version,last_event_time_in_utc,last_ids, andnext_link.The collector will automatically map old key names (e.g.
last_polled_timestampālast_event_time_in_utc) to the appropriate value.
Change log
Release | Released on | Release type | Details | Recommendations |
|---|---|---|---|---|
| Nov 29, 2024 | IMPROVEMENT SECURITY | Changed
Security
|
|
| Aug 2, 2024 | IMPROVEMENTBUG FIX | Improvements
Bug Fixing
|
|
| Apr 1, 2024 | IMPROVEMENT | Improvements
|
|
| Oct 18, 2023 | NEW FEATUREIMPROVEMENTBUG FIX | New features
Improvements
Bug fixing
|
|
| Ā Oct 10, 2023 | NEW FEATUREIMPROVEMENTBUG FIX | New features:
Improvements:
Bug fixing:
|
|
| Jan 14, 2023 | BUG FIXIMPROVEMENT | Improvements:
Bug fixing:
|
|
| Aug 2, 2022 | NEW FEATURE | New features:
Improvements:
|
|
| Dec 27, 2022 | BUG FIX | Fixed bugs:
|
|
| Dec 2, 2022 | BUG FIX | Fixed bugs:
|
|
| Dec 2, 2022 | IMPROVEMENT | Improvements:
|
|
| Nov 18, 2022 | IMPROVEMENTBUG FIX | Improvements:
Bug fixing:
|
|
| Aug 2, 2022 | NEW FEATURE IMPROVEMENT | New features:
Improvements:
|
|