Note |
---|
If you are migrating to version 2.X from version any 1.X x version, please read this section. |
Overview
...
API limits, delays and known issues
Microsoft SLAs can be anywhere from 3 minutes to 6 hours in most cases. Check more information here.
Vendor setup
...
Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:
An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.
You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.
Note |
---|
You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits. |
...
Action
The Graph API imposes some throttling limits to the applications. The number of requests in a given time is limited according to several criteria. There is a generic limitation for the Graph API and service specific limits.
Any request can be evaluated against multiple limits, depending on the scope of the limit (per app across all tenants, per tenant for all apps, per app per tenant, and so on), the request type (GET, POST, PATCH, and so on), and other factors. The first limit to be reached triggers throttling behavior.
In the case of the services used by the collector, the limits are:
Request type | Per app across all tenants |
---|
Any | 130,000 requests per 10 seconds |
provisioning_audits
| 5 requests per 10 seconds |
directory_audits
| 5 requests per 10 seconds |
signins
| 5 requests per 10 seconds |
signIns_v2
| 5 requests per 10 seconds |
alerts
| 150 requests per minute |
alerts_v2
| 150 requests per minute |
secure_scores
| 150 requests per minute or 10,000 in a 10-minute period |
secure_score_control_profiles
| 10,000 in a 10-minute period |
A complete reference of the limits can be found here.
When the collector reaches the API limit, the API returns a 429 error code. This is reflected in the logs of the collector with a message like 429 Error during API call
. If several of these messages are found, the solution is to use the request_period_in_seconds
for this service to decrease the frequency of calls.
For instance, a request period of 300 means that the API will be called every 5 minutes for this service, instead of the default value of 60 seconds.
API delays
Microsoft SLAs can be anywhere from 3 minutes to 6 hours in most cases. Check more information here.
Vendor setup
Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:
An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.
You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.
Note |
---|
You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits. |
| Action | Steps |
---|
1 | Register and configure the application | Go to Azure portal and click on Azure Active Directory. Click on App registration on the left-menu side. Then click on + New registration. On the Register and Application page: Name the application. Select Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) in Supported Accounts type. In Redirect URI (optional) leave it as default (blank). Click Register.
App registration page will open. Click on your app to configure it and give it permissions. You will see your app’s dashboard with information (docs, endpoints, etc.) when clicking it. Click Authentication on the left-menu side, then choose + Add a platform and select Mobile and desktop application. Select the three redirects URIs: https://login.microsoftonline.com/common/oauth2/nativeclient https://login.live.com/oauth20_desktop.srf msale36f3a02-3eef-437b-874e-8a0aa29a2bf0://auth
Click Configure.
|
---|
2 | Grant the required permissions | Go to API permissions on the left-menu side. Click + Add permission in case you don’t have Microsoft Graph in the API/Permission list. Select Application permissions and check SecurityEvents.Read.All . Check the following permissions: AuditLog.Read.All ,Directory.Read.All and User.Read.All . If you did everything correctly, permissions will display. Select Grant admin consent for the applications.
Info |
---|
You do not need to activate permissions if you are not going to use its corresponding resource. Check the Permissions reference per service section for a detailed breakdown on resource and their needed permissions. |
|
---|
3 | Obtain the requires credentials for the collector | Go to Certificates & Secrets, select + New client secret . Named it and copy the token value. Go to Overview to get your Tenant ID and Client ID and copy both values.
Note |
---|
The token will display only once. You will need to create another one if you didn’t copy it the first time. |
|
---|
...
Rw ui tabs macro |
---|
Rw tab |
---|
title | On-premise collector |
---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector: Code Block |
---|
<any_directory>
└── devo-collectors/
└── <product_name>/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── state/
└── config/
└── config.yaml |
Note |
---|
Replace <product_name> with the proper value. |
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/ . Learn more about security credentials in Devo here. Note |
---|
Replace <product_name> with the proper value. |
Editing the config.yaml file Code Block |
---|
globals:
debug: false
id: <collector_id_value>
name: <collector_name_value>
persistence:
type: filesystem
config:
directory_name: state
outputs:
devo_us_1:
type: devo_platform
config:
address: <devo_address>
port: 443
type: SSL
chain: <chain_filename>
cert: <cert_filename>
key: <key_filename>
inputs:
microsoft_graph_audit:
id: <short_unique_id>
enabled: true
credentials:
tenant_id: <tenant_id_value>
client_id: <client_id_value>
client_secret: <client_secret_value>
environment: <environment_value>
override_top_level_domain: <override_top_level_domain_value>
override_base_url: <override_base_url_value>
override_login_url: <override_login_url_value>
override_scope_value: <override_scope_value_value>
services:
directory_audits:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
provisioning_audits:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
signIns:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
signIns_v2:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
microsoft_graph_security:
id: <short_unique_id>
enabled: true
credentials:
tenant_id: <tenant_id_value>
client_id: <client_id_value>
client_secret: <client_secret_value>
environment: <environment_value>
override_top_level_domain: <override_top_level_domain_value>
override_base_url: <override_base_url_value>
override_login_url: <override_login_url_value>
override_scope_value: <override_scope_value_value>
services:
secure_score_control_profiles:
request_period_in_seconds: <request_period_in_seconds_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
secure_scores:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
alerts:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value>
alerts_v2:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
additional_filter: <additional_filter_value>
override_tag: <override_tag_value>
override_query_window_max_seconds: <override_query_window_max_seconds_value> |
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Replace the placeholders with your required values following the description table below: Parameter | Data type | Type | Value Range / Format | Details |
---|
collector_id_value
| str
| mandatory | minimum length: 1 maximum length: 5 | Use this parameter to give a unique ID to this collector. | collector_name_value
| str
| mandatory | minimum length: 1 maximum length: 10 | Use this parameter to give a name to this collector. | devo_address
| str
| mandatory | One of: collector-us.devo.io
collector-eu.devo.io
| Use this parameter to identify the Devo Cloud where the events will be sent. | chain_filename
| str
| mandatory | minimum length: 4 maximum length: 20 | Use this parameter to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt . | cert_filename
| str
| mandatory | minimum length: 4 maximum length: 20 | Use this parameter to identify the file.cert downloaded from your Devo domain. | key_filename
| str
| mandatory | minimum length: 4 maximum length: 20 | Use this parameter to identify the file.key downloaded from your Devo domain. | short_unique_id
| str
| mandatory | minimum length: 5 maximum length: - | Use this parameter to give a unique ID to this input service. This parameter is used to build the persistence address; do not use the same value for multiple collectors. It could cause a collision. | tenant_id_value
| str
| mandatory | minimum length: 1 | This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | client_id_value
| str
| mandatory | minimum length: 1 | This is the Application (client) ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | client_secret_value
| str
| mandatory | minimum length: 1 | This is the client secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. | environment_value
| str
| optional | minimum length: 1 | This is an optional control parameter that is injected into the events and allows you to differentiate the environment. For example: dev and prod. Info |
---|
This parameter should be removed if it is not used. |
| request_period_in_seconds_value
| int
| optional | minimum length: 60 | Period in seconds used between each data pulling. This value will overwrite the default value (300 seconds). | start_time_in_utc_value
| str
| optional | UTC datetime string having datetime string format %Y-%m-%dT%H-%M-%SZ (e.g., “2020-01-01T00:00:01Z”) | This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events. Info |
---|
This parameter should be removed if it is not used. |
| additional_filter_value
| str
| optional | minimum length: 1 | This parameter allows you to add an additional Microsoft Graph OData query clause to the default filters. By default, the collector attempts to fetch all data possible for a given Microsoft Graph entity type. See OData query parameters documentation for more information on how to format filters for Microsoft Graph API. Info |
---|
This parameter should be removed if it is not used. |
| override_tag_value
| str
| optional | Devo tag-friendly string (no special characters, spaces, etc.) For more information see Devo Tags | An optional tag that allows users to override the service default tags. Info |
---|
This parameter should be removed if it is not used. |
| override_top_level_domain_value
| str
| optional | One of: Default value:com | This is an optional parameter that allows you to override the default top level domain. If you're working in a Microsoft 365 GCC environment, continue using the worldwide endpoints (com ). If you're working in a Microsoft 365 GCC High environment, use us . Info |
---|
This parameter should be removed if it is not used. |
| override_base_url_value
| str
| optional | default value: https://graph.microsoft.{top_level_domain} | A parameter that allows you to override the base URL. Info |
---|
This parameter should be removed if it is not used. |
| override_login_url_value
| str
| optional | default value: https://login.microsoftonline.com/ | A parameter that allows you to override the login URL. Info |
---|
This parameter should be removed if it is not used. |
| override_scope_value
| str
| optional | default value: https://graph.microsoft.com/.default | A parameter that allows you to override the scope value. Info |
---|
This parameter should be removed if it is not used. |
| override_query_window_max_seconds
| int
| optional | default value: None | A parameter that allows you to override the maximum query window. This is useful for any APIs that greatly restrict the amount of data one can fetch for a given datetime range (performance reasons). The collector will continuously fetch data but in smaller increments. Info |
---|
This parameter should be removed if it is not used. |
|
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: Use the following command to add the Docker image to the system: Code Block |
---|
gunzip -c <collector-ms-graph-collector-if-docker-image>-<1.2.0>.tgz | docker load |
Note |
---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value. |
The Docker image can be deployed on the following services: DockerExecute the following command on the root directory <any_directory>/devo-collectors/<product_name>/ Code Block |
---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
Note |
---|
Replace <product_name> , <image_name> and <version> with the proper values. |
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory. Code Block |
---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory: Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace <product_name> , <image_name> and <version> with the proper values. |
We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer: In the Collector Server GUI, access the domain in which you want this instance to be created Click Add Collector and find the one you wish to add. In the Version field, select the latest value. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain). In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups . In the Parameters section, establish the Collector Parameters as follows below:
Editing the JSON configuration Code Block |
---|
{
"global_overrides": {
"debug": false
},
"inputs": {
"microsoft_graph_audit": {
"id": "<short_unique_id>",
"enabled": true,
"credentials": {
"tenant_id": "<tenant_id_value>",
"client_id": "<client_id_value>",
"client_secret": "<client_secret_value>"
},
"environment": "<environment_value>",
"override_top_level_domain": "<override_top_level_domain_value>",
"override_base_url": "<override_base_url_value>",
"override_login_url": "<override_login_url_value>",
"override_scope_value": "<override_scope_value_value>",
"services": {
"directory_audits": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"provisioning_audits": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"signIns": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"signIns_v2": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
}
}
},
"microsoft_graph_security": {
"id": "<short_unique_id>",
"enabled": true,
"credentials": {
"tenant_id": "<tenant_id_value>",
"client_id": "<client_id_value>",
"client_secret": "<client_secret_value>"
},
"environment": "<environment_value>",
"override_top_level_domain": "<override_top_level_domain_value>",
"override_base_url": "<override_base_url_value>",
"override_login_url": "<override_login_url_value>",
"override_scope_value": "<override_scope_value_value>",
"services": {
"secure_score_control_profiles": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>"
},
"secure_scores": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"alerts": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
},
"alerts_v2": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"additional_filter": "<additional_filter_value>",
"override_tag": "<override_tag_value>",
"override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
}
}
}
}
} |
Parameter | Data type | Type | Value Range / Format | Details |
---|
short_unique_id
| str
| mandatory | minimum length: 1 maximum length: 5 | Use this parameter to give a unique ID to this input service. This parameter is used to build the persistence address; do not use the same value for multiple collectors. It could cause a collision. | tenant_id_value
| str
| mandatory | minimum length: 1 | This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | client_id_value
| str
| mandatory | minimum length: 1 | This is the Application (client) ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | client_secret_value
| str
| mandatory | minimum length: 1 | This is the client secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. | environment_value
| str
| optional | minimum length: 1 | This is an optional control parameter that is injected into the events and allows you to differentiate the environment. For example: dev and prod. Info |
---|
This parameter should be removed if it is not used. |
| request_period_in_seconds_value
| int
| optional | minimum length: 60 | Period in seconds used between each data pulling. This value will overwrite the default value (300 seconds). | start_time_in_utc_value
| str
| optional | UTC datetime string having datetime string format %Y-%m-%dT%H-%M-%SZ (e.g., “2020-01-01T00:00:01Z”) | This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events. Info |
---|
This parameter should be removed if it is not used. |
| additional_filter_value
| str
| optional | minimum length: 1 | This parameter allows you to add an additional Microsoft Graph OData query clause to the default filters. By default, the collector attempts to fetch all data possible for a given Microsoft Graph entity type. See OData query parameters documentation for more information on how to format filters for Microsoft Graph API. Info |
---|
This parameter should be removed if it is not used. |
| override_tag_value
| str
| optional | Devo tag-friendly string (no special characters, spaces, etc.) For more information see Devo Tags | An optional tag that allows users to override the service default tags. Info |
---|
This parameter should be removed if it is not used. |
| override_top_level_domain_value
| str
| optional | One of: Default value:com | This is an optional parameter that allows you to override the default top level domain. If you're working in a Microsoft 365 GCC environment, continue using the worldwide endpoints (com ). If you're working in a Microsoft 365 GCC High environment, use us . Info |
---|
This parameter should be removed if it is not used. |
| override_base_url_value
| str
| optional | default value: https://graph.microsoft.{top_level_domain} | A parameter that allows you to override the base URL. Info |
---|
This parameter should be removed if it is not used. |
| override_login_url_value
| str
| optional | default value: https://login.microsoftonline.com/ | A parameter that allows you to override the login URL. Info |
---|
This parameter should be removed if it is not used. |
| override_scope_value
| str
| optional | default value: https://graph.microsoft.com/.default | A parameter that allows you to override the scope value. Info |
---|
This parameter should be removed if it is not used. |
| override_query_window_max_seconds
| int
| optional | default value: None | |
|
...
Error type | Error ID | Error message | Cause | Solution |
---|
InitVariablesError
| 1 | Invalid start_time_in_utc: {ini_start_str} . Must be in parseable datetime format. | The configured start_time_in_utc parameter is a non-parseable format. | Update the start_time_in_utc value to have the recommended format as indicated in the guide. |
InitVariablesError
| 2 | Invalid start_time_in_utc: {ini_start_str} . Must be in the past. | The configured start_time_in_utc parameter is a future date. | Update the start_time_in_utc value to a past datetime. |
SetupError
| 101 | Failed to fetch OAuth token from {token_endpoint} . Exception: {e} . | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
SetupError
| 102 | Failed to fetch data from {endpoint} . Source is not pullable. | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
ApiError
| 401 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 401 response. | Ensure that the provided credentials are correct and provide read access to the targeted data. |
ApiError
| 429 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 429 response. The collector will attempt to retry requests (default up to 3 times) and respect back-off headers if they exist. If the collector repeatedly encounters this error, adjust the rate limit and/or contact the API provider to ensure that you have enough quota to complete the data pull. Please note that IBM Cloud supports a limited amount of concurrent searches for the Activity Tracker instances at any given time. Please refer to your IBM Cloud plan for these limitations (most plans only support one concurrent search). If you encounter a 429 due to a concurrent search, and the collector proceeds normally during the subsequent pull, then there is nothing to correct. If you repeatedly encounter a 429 due to a concurrent search, make sure that you do not have multiple collectors running against the same Activity Tracker instance and that you do not have other, non-Devo-collector API searches running against the Activity Tracker instance.returned an HTTP 429 response. | The collector will attempt to retry requests (default up to 3 times) and respect back-off headers if they exist. If the collector repeatedly encounters this error, adjust the request_period_in_seconds . (see section about API limits) |
ApiError
| 498 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 500 response. | If the API returns a 500 but successfully completes subsequent runs then you may ignore this error. If the API repeatedly returns a 500 error, ensure the server is reachable and operational. |
...