Table of Contents | ||||
---|---|---|---|---|
|
Service description
Microsoft Graph provides many services such as Microsoft 365, Office 365, Outlook, and others. At this moment, the Microsoft Graph collector only deals with security alerts and scores retrieved from the Microsoft products. This empowers customers to streamline security operations and better defend against increasing cyber threats. The Microsoft Graph collector includes the two key entities described in the following sections:
...
Table of Contents | ||||
---|---|---|---|---|
|
Note |
---|
If you are migrating from any 1.x version, please read this section. |
Overview
The Microsoft Graph Collector provides the ability to collect data and intelligence from services such as Microsoft 365, Windows, and Enterprise Mobility and Security. Currently, this data collector is able to ingest only security alerts, scores, provisioning, audit and sign ins retrieved from the Microsoft products. This empowers customers to streamline security operations and better defend against increasing cyber threats faced in their Azure AD and Microsoft 365 environments and beyond.
Devo’s Microsoft Graph collector also enables customers to correlate events and context to improve threat protection and response, and includes key entities described in the next sections.
Configuration requirements
To run this collector, there are some configurations detailed below that you need to consider.
Configuration | Details |
---|---|
Azure account | Azure account with admin level permissions and Azure AD tenant. |
Credentials | The credentials configuration block has been filled correctly. |
Info |
---|
More information Refer to the Vendor setup section to know more about these configurations. |
Devo collector features
Feature | Details |
---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Allowed source events obfuscation |
|
Data sources
Data source | Description | API endpoint | Collector service name | Devo table |
---|---|---|---|---|
Audit logs - provisioning | Represents an action performed by the Microsoft Entra provisioning service and its associated properties. |
|
|
|
Audit logs - directory | Represents the directory audit items and its collection. |
|
|
|
Audit logs - sign-ins | Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API. |
|
|
|
Audit logs - sign-ins (v2) | Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API. |
|
|
|
Legacy Alerts | This resource corresponds to the first generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft or a partner security |
...
Application name
Details
Devo data tables
alerts
solution has identified. This type of alerts federates calling of supported Azure and Microsoft 365 Defender security providers listed in Use the Microsoft Graph security API. It aggregates common alert data among the different domains to allow applications to unify and streamline management of security issues across all integrated solutions. |
Alerts Security Providers:
Azure Security Center
Azure Active Directory Identity Protection
Microsoft Cloud App Security
Microsoft Defender Advanced Threat Protection
Azure Advanced Threat Protection
Cloud App Security (Update coming soon from MS Graph)
Azure Information Protection
Azure Sentinel
...
Microsoft Secure Score is a security analytics solution that gives you visibility into your security portfolio and how to improve it. With a single score, you can better understand what you have done to reduce your risk in Microsoft solutions. You can also compare your score with other organizations and see how your score has been trending over time.
The Microsoft Graph secure score and secureScoreControlProfile entities help you balance your organization's security and productivity needs while enabling the appropriate mix of security features. You can also project what your score would be after you adopt security features.
Data source description
Currently, the Microsoft Graph collector generates security activities for these resources. The collector processes the Microsoft Graph responses and sends them to the Devo platform, which will categorize all the information received on tables along rows and columns on your Devo domain.
Microsoft Graph resources
Listed in the table below are the application names, details, and how the Devo platform treats the data and to which tables sends it:
|
|
| ||
Alerts (v2) | This resource corresponds to the latest generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft 365 Defender, or a security provider integrated with Microsoft 365 Defender, has identified. When detecting a threat, a security provider creates an alert in the system. Microsoft 365 Defender pulls this alert data from the security provider, and consumes the alert data to return valuable clues in an alert resource about any related attack, impacted assets, and associated evidence. It automatically correlates other alerts with the same attack techniques or the same attacker into an incident to provide a broader context of an attack. Aggregating alerts in this manner makes it easy for analysts to collectively investigate and respond to threats. |
|
|
|
| |
Secure Scores | Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held. |
cloud.msgraph.security.scores
secureScoreControlProfile
This data is sorted by createdDateTime, from latest to earliest. This will allow you to page responses by using $top=n, where n = the number of days of data that you want to retrieve. |
|
|
| |
Secure Scores Control Profiles | Represents a tenant's secure score per control data. By default, this resource returns all controls for a tenant and can explicitly pull individual controls. |
|
|
|
|
Info |
---|
For more info about Microsoft Graph API, visit Microsoft Graph Reference. |
Setup
The Microsoft Graph data collector works over the Microsoft products, such as Microsoft Azure Directory. To active the alerts and secure score resources from the Microsoft Graph API, a subscription on Microsoft Azure Directory followed by an app registration should be created, as well as configuring the resources with the right permissions for the best performance of the collector.
Setting up permissions on the subscription
Go to the Azure portal and click Azure Activity Directory.
Click App registrations → New registration to create a new app.
On the Register an Application page, give your application a name.
On Supported Accounts Type, select the third option (Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) )
On Redirect URI (optional), leave it blank (as default) and click Register.
After registering the app, it will be displayed in a list on the App registration page. Click your app to give it permissions and configure it. You’ll see the app on the dashboard with some important information, docs, and endpoints.
On the left menu, click Authentication → Add a platform → Mobile and desktop applications.
Mark the 3 redirect URIs and click configure.
On the left menu, click API permissions and check if you already have Microsoft Graph on the API/ Permission list. If not, click Add permission and add Microsoft Graph.
Now select Application permissions and search for Security. Check all the boxes available for the service. Then, repeat the same process with Audit and User. If you have done everything correctly, your permissions will display as shown on the green box line. Then Grant admin consent for the applications.
Note |
---|
Troubleshooting If you get this error “Unable to save changes. One or more of the following permission(s) are currently not supported: SecurityEvents.ReadWrite.All, SecurityEvents.Read.All, SecurityActions.Read.All, SecurityActions.ReadWrite.All. Please remove these permission(s) and retry your request. [O6b9]” you might not have set up the permission correctly. Make sure that your configuration is exactly the same as in the green box in the capture above. |
Authentication
After applying the permissions, select Certificates & secrets → New client secret, enter the desired name, and copy the token.
Note |
---|
The token will display just once. You might have to create another in case you don’t copy it. |
...
Getting the credentials
After creating the token, go to Overview to get your Tenant ID and Client ID. This information will be used on the collector server to run the application.
...
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
...
Rw tab | ||
---|---|---|
|
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.
Rw tab | ||
---|---|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
...
Code Block |
---|
<any_directory>
└── devo-collectors/
└── msgraph/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
└── config/
└── config-msgraph.yaml |
...
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/msgraph/certs
. Learn more about security credentials in Devo here.
...
In the config-msgraph.yaml file, replace the <short_unique_identifier>
, <tenant_id_value>
, <client_id_value>
, and <client_secret_value>
values and enter the ones that you got in the previous steps. See inline comments to better understand the file.
...
|
API limits, delays and known issues
The Graph API imposes some throttling limits to the applications. The number of requests in a given time is limited according to several criteria. There is a generic limitation for the Graph API and service specific limits.
Any request can be evaluated against multiple limits, depending on the scope of the limit (per app across all tenants, per tenant for all apps, per app per tenant, and so on), the request type (GET, POST, PATCH, and so on), and other factors. The first limit to be reached triggers throttling behavior.
In the case of the services used by the collector, the limits are:
Request type | Per app across all tenants |
---|---|
Any | 130,000 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 150 requests per minute |
| 150 requests per minute |
| 150 requests per minute or 10,000 in a 10-minute period |
| 10,000 in a 10-minute period |
A complete reference of the limits can be found here.
When the collector reaches the API limit, the API returns a 429 error code. This is reflected in the logs of the collector with a message like 429 Error during API call
. If several of these messages are found, the solution is to use the request_period_in_seconds
for this service to decrease the frequency of calls.
For instance, a request period of 300 means that the API will be called every 5 minutes for this service, instead of the default value of 60 seconds.
API delays
Microsoft SLAs can be anywhere from 3 minutes to 6 hours in most cases. Check more information here.
Vendor setup
Anchor | ||||
---|---|---|---|---|
|
Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:
An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.
You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.
Note |
---|
You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits. |
Action | Steps | |||
1 | Register and configure the application |
| ||
2 | Grant the required permissions |
| ||
3 | Obtain the requires credentials for the collector |
|
Permission reference per service
Collector service | Resource | Required permissions | Microsoft documentation |
---|---|---|---|
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
Required for all services |
|
|
Note |
---|
Troubleshooting Sometimes you’ll see this error: Unable to save changes. One or more of the following permission(s) are currently not supported: SecurityEvents.Read.All or SecurityActions.Read.All. Please remove these permission(s) and retry your request. [O6b9]. It might that you did not set up the permission correctly. Please, make sure that the permissions are exactly are showing above. |
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
Info |
---|
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details. |
Setting | Details | ||
---|---|---|---|
| This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | ||
| This is the Client’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.
| ||
| This is the Client’s secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. |
Info |
---|
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method. |
Accepted authentication methods
This collector only accepts one single authentication method. You will have to fill the following properties on the credentials
configuration block:
Authentication method | Tenant ID | Client ID | Client secret | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Rw ui tabs macro | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector:
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in
Editing the config.yaml file
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: | ||||||||||||||||
Collector Docker image | SHA-256 hash | |||||||||||||||
|
Code Block |
---|
gunzip -c collector-msgraph-docker-image-<version>.tgz | docker load |
Info |
---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace " |
The Docker image can be deployed on the following services:
Docker
Docker Compose
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/msgraph/
Code Block |
---|
docker run \
--name collector-msgraph \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-msgraph.yaml \
--rm -it docker.devo.internal/collector/msgraph:<version> |
Note |
---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/msgraph/
directory.
Code Block |
---|
version: '3'
services:
collector-msgraph:
build:
context: .
dockerfile: Dockerfile
image: docker.devo.internal/collector/msgraph:${IMAGE_VERSION:-latest}
container_name: collector-msgraph
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
environment:
- CONFIG_FILE=${CONFIG_FILE:-config-msgraph.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/msgraph/
directory:
Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace <version> with a proper value.<short_unique_id>
enabled: true
credentials:
tenant_id: <tenant_id_value>
client_id: <client_id_value>
client_secret: <client_secret_value>
environment: <environment_value>
override_top_level_domain: <override_top_level_domain_value>
override_base_url: <override_base_url_value>
override_login_url: <override_login_url_value>
override_scope_value: <override_scope_value_value>
services:
directory_audits:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
provisioning_audits:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
signIns:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
signIns_v2:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
microsoft_graph_security:
id: <short_unique_id>
enabled: true
credentials:
tenant_id: <tenant_id_value>
client_id: <client_id_value>
client_secret: <client_secret_value>
environment: <environment_value>
override_top_level_domain: <override_top_level_domain_value>
override_base_url: <override_base_url_value>
override_login_url: <override_login_url_value>
override_scope_value: <override_scope_value_value>
services:
secure_score_control_profiles:
request_period_in_seconds: <request_period_in_seconds_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
secure_scores:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
alerts:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
alerts_v2:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value> |
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the |
Replace the placeholders with your required values following the description table below:
Parameter | Data type | Type | Value Range / Format | Details | ||
---|---|---|---|---|---|---|
|
| mandatory | minimum length: 1 | Use this parameter to give a unique ID to this collector. | ||
|
| mandatory | minimum length: 1 | Use this parameter to give a name to this collector. | ||
|
| mandatory | One of:
| Use this parameter to identify the Devo Cloud where the events will be sent. | ||
|
| mandatory | minimum length: 4 | Use this parameter to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: | ||
|
| mandatory | minimum length: 4 | Use this parameter to identify the | ||
|
| mandatory | minimum length: 4 | Use this parameter to identify the | ||
|
| mandatory | minimum length: 5 | Use this parameter to give a unique ID to this input service. | ||
|
| mandatory | minimum length: 1 | This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | ||
|
| mandatory | minimum length: 1 | This is the Application (client) ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | ||
|
| mandatory | minimum length: 1 | This is the client secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. | ||
|
| optional | minimum length: 1 | This is an optional control parameter that is injected into the events and allows you to differentiate the environment. For example: dev and prod.
| ||
|
| optional | minimum length: 60 | Period in seconds used between each data pulling. This value will overwrite the default value (300 seconds). | ||
|
| optional | UTC datetime string having datetime string format | This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events.
| ||
|
| optional | minimum length: 1 | This parameter allows you to add an additional Microsoft Graph OData query clause to the default filters. By default, the collector attempts to fetch all data possible for a given Microsoft Graph entity type. See OData query parameters documentation for more information on how to format filters for Microsoft Graph API.
| ||
|
| optional | Devo tag-friendly string (no special characters, spaces, etc.) For more information see Devo Tags | An optional tag that allows users to override the service default tags.
| ||
|
| optional | One of:
Default value: | This is an optional parameter that allows you to override the default top level domain. If you're working in a Microsoft 365 GCC environment, continue using the worldwide endpoints ( If you're working in a Microsoft 365 GCC High environment, use
| ||
|
| optional | default value: | A parameter that allows you to override the base URL.
| ||
|
| optional | default value: | A parameter that allows you to override the login URL.
| ||
|
| optional | default value: | A parameter that allows you to override the scope value.
| ||
|
| optional | default value: None | A parameter that allows you to override the maximum query window. This is useful for any APIs that greatly restrict the amount of data one can fetch for a given datetime range (performance reasons). The collector will continuously fetch data but in smaller increments.
|
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Collector Docker image | SHA-256 hash |
---|---|
|
Use the following command to add the Docker image to the system:
Code Block |
---|
gunzip -c <collector-ms-graph-collector-if-docker-image>-<1.2.0>.tgz | docker load |
Note |
---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace |
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Code Block |
---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
Note |
---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/
directory.
Code Block |
---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/
directory:
Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace |
Rw tab | ||
---|---|---|
|
We use a piece of software called Collector Server to host and manage all our available collectors.
To enable the collector for a customer:
In the Collector Server GUI, access the domain in which you want this instance to be created
Click Add Collector and find the one you wish to add.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the sending method select Direct Send. Direct Send configuration is optional for collectors that create
Table
events, but mandatory for those that createLookups
.In the Parameters section, establish the Collector Parameters as follows below:
Editing the JSON configuration
Code Block |
---|
{
"global_overrides": {
"debug": false
},
"inputs": {
"microsoft_graph_audit": {
"id": "<short_unique_id>",
"enabled": true,
"credentials": {
"tenant_id": "<tenant_id_value>",
"client_id": "<client_id_value>",
"client_secret": "<client_secret_value>"
},
"environment": "<environment_value>",
"override_top_level_domain": "<override_top_level_domain_value>",
"override_base_url": "<override_base_url_value>",
"override_login_url": "<override_login_url_value>",
"override_scope_value": "<override_scope_value_value>",
"services": {
"directory_audits": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"provisioning_audits": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"signIns": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"signIns_v2": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
}
}
},
"microsoft_graph_security": {
"id": "<short_unique_id>",
"enabled": true,
"credentials": {
"tenant_id": "<tenant_id_value>",
"client_id": "<client_id_value>",
"client_secret": "<client_secret_value>"
},
"environment": "<environment_value>",
"override_top_level_domain": "<override_top_level_domain_value>",
"override_base_url": "<override_base_url_value>",
"override_login_url": "<override_login_url_value>",
"override_scope_value": "<override_scope_value_value>",
"services": {
"secure_score_control_profiles": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>"
},
"secure_scores": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"alerts": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"alerts_v2": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
}
}
}
}
} |
Parameter | Data type | Type | Value Range / Format | Details | ||
---|---|---|---|---|---|---|
|
| mandatory | minimum length: 1 | Use this parameter to give a unique ID to this input service. | ||
|
| mandatory | minimum length: 1 | This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | ||
|
| mandatory | minimum length: 1 | This is the Application (client) ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | ||
|
| mandatory | minimum length: 1 | This is the client secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. | ||
|
| optional | minimum length: 1 | This is an optional control parameter that is injected into the events and allows you to differentiate the environment. For example: dev and prod.
| ||
|
| optional | minimum length: 60 | Period in seconds used between each data pulling. This value will overwrite the default value (300 seconds). | ||
|
| optional | UTC datetime string having datetime string format | This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events.
| ||
|
| optional | minimum length: 1 | This parameter allows you to add an additional Microsoft Graph OData query clause to the default filters. By default, the collector attempts to fetch all data possible for a given Microsoft Graph entity type. See OData query parameters documentation for more information on how to format filters for Microsoft Graph API.
| ||
|
| optional | Devo tag-friendly string (no special characters, spaces, etc.) For more information see Devo Tags | An optional tag that allows users to override the service default tags.
| ||
|
| optional | One of:
Default value: | This is an optional parameter that allows you to override the default top level domain. If you're working in a Microsoft 365 GCC environment, continue using the worldwide endpoints ( If you're working in a Microsoft 365 GCC High environment, use
| ||
|
| optional | default value: | A parameter that allows you to override the base URL.
| ||
|
| optional | default value: | A parameter that allows you to override the login URL.
| ||
|
| optional | default value: | A parameter that allows you to override the scope value.
| ||
|
| optional | default value: None |
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll directory audit records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll provisioning audit records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll sign-in records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
Internal process and deduplication methodAll sign-in_v2 records are continuously pulled subject to the Devo categorization and destinationEvents of this service are ingested into the following tables:
|
Expand | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||
Internal process and deduplication methodAll alert records are continuously pulled subject to the Devo categorization and destinationEvents of this service are ingested into the following tables:
|
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll alerts_v2 records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll secure score records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll secure score profile records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Restart the persistence for a service
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc
parameter to a different one.Save the changes.
Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Common logic
Error type | Error ID | Error message | Cause | Solution |
---|---|---|---|---|
| 1 | Invalid | The configured | Update the |
| 2 | Invalid | The configured | Update the |
| 101 | Failed to fetch OAuth token from | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
| 102 | Failed to fetch data from | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
| 401 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 401 response. | Ensure that the provided credentials are correct and provide read access to the targeted data. |
| 429 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 429 response. | The collector will attempt to retry requests (default up to 3 times) and |
| 498 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 500 response. | If the API returns a 500 but successfully completes subsequent runs then you may ignore this error. If the API repeatedly returns a 500 error, ensure the server is reachable and operational. |
Collector operations
Verify collector operations
Expand | ||
---|---|---|
| ||
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module:
|
Expand | ||
---|---|---|
| ||
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module:
|
Expand | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
The Integrations Factory Collector SDK has 3 different sender services depending on the event type to deliver (internal, standard, and lookup). This collector uses the following Sender Services:
|
Expand | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
|
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
Code Block |
---|
INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB ->
410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB ->
705.28MiB)Change log |
Anchor | ||||
---|---|---|---|---|
|
1.X to 2.X migration guide
Version 2.0.0 introduced several changes to the collector's configuration. When upgrading the collector, users must make changes to their configuration file to ensure that the collector continues to work as expected.
Note |
---|
Important Note that |
In versions 1.X of the collector, some services had a config parameter tag_version
with values v1
or v2
. The effect of this parameter is that the destination table for these services will be different. These are the destination tables according to the tag_version
value:
v1 | v2 (default) |
---|---|
|
|
|
|
In the new collector 2.0.0, the old config parameter tag_version
has been removed. The same effect as v1
can be made using override_tag
, with these values:
Service |
|
---|---|
|
|
|
|
For instance, if we have this in the old config:
Code Block |
---|
secure_scores:
tag_version: v1
secure_score_control_profile:
tag_version: v1 |
you should change it to:
Code Block |
---|
secure_scores:
override_tag: cloud.msgraph.security.scores.2
secure_score_control_profile:
override_tag: cloud.msgraph.security.scorecontrol.2 |
Note |
---|
Important Note that security alerts are sent to different tables according to their categories. |
In old versions of this collector, all alerts were sent to the table cloud.msgraph.security.alerts
However, as there are alerts of several types, in version 2.0.0 alerts are categorized according to their vendorInformation-provider
field (their vendor) and sent to different tables:
Vendor | Old table | New table |
---|---|---|
IPC |
|
|
MCAS |
| |
Microsoft Defender ATP |
| |
Microsoft 365 Defender |
| |
Office 365 Security and Compliance |
| |
Azure Sentinel |
| |
ASC |
| |
Azure Advanced Threat Protection |
| |
Others |
|
It is possible to avoid this feature and send all alerts to the same table by editing the config file and changing the tag:
Code Block |
---|
alerts:
override_tag: cloud.msgraph.security.alerts.2 |
Config file comparison
Below you will find a sample 1.7.1 configuration file and a sample 2.0.0 configuration file.
Expand | ||
---|---|---|
| ||
|
Expand | ||
---|---|---|
| ||
|
Config file changes
The URL endpoints (
override_base_url_main
,override_base_url_vendor
,override_base_url_vendor_with_sub_provider
,override_login_url
) have been moved from the individual services to the global configuration section.override_base_url_main
has been renamed tooverride_base_url
.tag_version
has been removed.pull_sliding_window_timespan_period
has been removed.reset_persistence_auth
has been removed.override_time_delta_in_days
has been removed.ms_365_environment
has been replaced byoverride_top_level_domain
. GCC High Gov should useus
in theoverride_top_level_domain
. Alternatively, users can useoverride_base_url
to specify the GCC High Gov base URL.additional_filter
has been added to all services. Users can use this field to specify additional filters that will be applied when querying the Microsoft Graph API.The collector can use new services from Graph (beta endpoint in Graph), that services, that use to be in a separate service for each type, have been consolidated into one service called
signIns_v2
. Users should remove all three services from their config and use only thesignIns_v2
service.start_time
has been renamed tostart_time_in_utc
.
Persistence changes
The persistence object consists of the following fields:
persistence_version
,last_event_time_in_utc
,last_ids
, andnext_link
.The collector will automatically map old key names (e.g.
last_polled_timestamp
→last_event_time_in_utc
) to the appropriate value.
Change log
Release | Released on | Release type | Details | Recommendations | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| Improvements
Bug Fixing
|
| |||||||||||||||||||
|
| Improvements
|
| |||||||||||||||||||
|
| New features
Improvements
Bug fixing
|
| |||||||||||||||||||
|
|
| New features:
Improvements:
Bug fixing:
|
| ||||||||||||||||||
|
| Improvements:
Bug fixing:
|
| |||||||||||||||||||
|
| New features:
Improvements:
|
| |||||||||||||||||||
|
| Fixed bugs:
|
| |||||||||||||||||||
|
| Fixed bugs:
|
| |||||||||||||||||||
|
| Improvements:
|
| |||||||||||||||||||
|
| Improvements:
Bug fixing:
|
| |||||||||||||||||||
|
| New features:
Improvements:
|
|