Page Comparison

Table of Contents

maxLevel	2
type	flat

Service description

Microsoft Graph provides many services such as Microsoft 365, Office 365, Outlook, and others. At this moment, the Microsoft Graph collector only deals with security alerts and scores retrieved from the Microsoft products. This empowers customers to streamline security operations and better defend against increasing cyber threats. The Microsoft Graph collector includes the two key entities described in the following sections:

Microsoft Graph collector#Alerts
Secure Scores

...

Table of Contents

maxLevel	2
type	flat

Note
If you are migrating from any 1.x version, please read this section.

Overview

The Microsoft Graph Collector provides the ability to collect data and intelligence from services such as Microsoft 365, Windows, and Enterprise Mobility and Security. Currently, this data collector is able to ingest only security alerts, scores, provisioning, audit and sign ins retrieved from the Microsoft products. This empowers customers to streamline security operations and better defend against increasing cyber threats faced in their Azure AD and Microsoft 365 environments and beyond.

Devo’s Microsoft Graph collector also enables customers to correlate events and context to improve threat protection and response, and includes key entities described in the next sections.

Configuration requirements

To run this collector, there are some configurations detailed below that you need to consider.

Configuration	Details
Azure account	Azure account with admin level permissions and Azure AD tenant.
Credentials	The credentials configuration block has been filled correctly.

Info

More information

Refer to the Vendor setup section to know more about these configurations.

Devo collector features

Feature	Details
Allow parallel downloading (`multipod`)	`not allowed`
Running environments	`collector server` `on-premise`
Populated Devo events	`table`
Flattening preprocessing	`no`
Allowed source events obfuscation	`yes`

Data sources

Data source	Description	API endpoint	Collector service name	Devo table
Audit logs - provisioning	Represents an action performed by the Microsoft Entra provisioning service and its associated properties.	`v1.0/auditLogs/provisioning`	`provisioning_audits`	`cloud.azure.ad.provisioning.*.msgraph`
Audit logs - directory	Represents the directory audit items and its collection.	`v1.0/auditLogs/directoryaudits`	`directory_audits`	`cloud.azure.ad.audit.*.msgraph`
Audit logs - sign-ins	Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API.	`v1.0/auditLogs/signIns`	`signIns`	`cloud.azure.ad.signin.*.msgraph`
Audit logs - sign-ins (v2)	Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API.	`beta/auditLogs/signIns`	`signIns_v2`	`cloud.azure.ad.interactive_user_signin..msgraph` `cloud.azure.ad.noninteractive_user_signin..msgraph` `cloud.azure.ad.managed_identity_signin..msgraph` `cloud.azure.ad.service_principal_signin..msgraph` `cloud.azure.ad.unknown_future_value_signin.*.msgraph`
Legacy Alerts	This resource corresponds to the first generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft or a partner security

...

Application name

Details

Devo data tables

alerts

Represents potential security issues within a customer's tenant that Microsoft or partner security solutions have identified. Use alerts to unify and streamline security issue management across all integrated solutions.

solution has identified.

This type of alerts federates calling of supported Azure and Microsoft 365 Defender security providers listed in Use the Microsoft Graph security API. It aggregates common alert data among the different domains to allow applications to unify and streamline management of security issues across all integrated solutions.

Alerts Security Providers:

Azure Security Center
Azure Active Directory Identity Protection
Microsoft Cloud App Security
Microsoft Defender Advanced Threat Protection
Azure Advanced Threat Protection
Cloud App Security (Update coming soon from MS Graph)
Azure Information Protection
Azure Sentinel

...

Microsoft Secure Score is a security analytics solution that gives you visibility into your security portfolio and how to improve it. With a single score, you can better understand what you have done to reduce your risk in Microsoft solutions. You can also compare your score with other organizations and see how your score has been trending over time.

The Microsoft Graph secure score and secureScoreControlProfile entities help you balance your organization's security and productivity needs while enabling the appropriate mix of security features. You can also project what your score would be after you adopt security features.

Data source description

Currently, the Microsoft Graph collector generates security activities for these resources. The collector processes the Microsoft Graph responses and sends them to the Devo platform, which will categorize all the information received on tables along rows and columns on your Devo domain.

Microsoft Graph resources

Listed in the table below are the application names, details, and how the Devo platform treats the data and to which tables sends it:

v1.0/security/alerts

alerts

cloud.azure.ad.alerts.*.msgraph
cloud.office365.cloud_apps.alerts.*.msgraph
cloud.office365.endpoint.alerts.*.msgraph
cloud.office365.security.alerts.*.msgraph
cloud.azure.sentinel.alerts.*.msgraph
cloud.office365.identity.alerts.*.msgraph
cloud.azure.securitycenter.alerts.*.msgraph

Alerts (v2)

This resource corresponds to the latest generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft 365 Defender, or a security provider integrated with Microsoft 365 Defender, has identified.

When detecting a threat, a security provider creates an alert in the system. Microsoft 365 Defender pulls this alert data from the security provider, and consumes the alert data to return valuable clues in an alert resource about any related attack, impacted assets, and associated evidence. It automatically correlates other alerts with the same attack techniques or the same attacker into an incident to provide a broader context of an attack. Aggregating alerts in this manner makes it easy for analysts to collectively investigate and respond to threats.

v1.0/security/alerts_v2

alerts_v2

cloud.msgraph.security.alerts

secureScore

`_v2.*`
Secure Scores	Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held.

cloud.msgraph.security.scores

secureScoreControlProfile

Represents a tenant's secure score per control data. By default, it

This data is sorted by createdDateTime, from latest to earliest. This will allow you to page responses by using $top=n, where n = the number of days of data that you want to retrieve.	`v1.0/security/secureScores`	`secure_scores`	`cloud.office365.security.scores.*.msgraph`
Secure Scores Control Profiles	Represents a tenant's secure score per control data. By default, this resource returns all controls for a tenant and can explicitly pull individual controls.	`v1.0/security/secureScoreControlProfiles/`	`secure_score_control_profiles`	`cloud.`

msgraph

office365.security.scorecontrol

Info
For more info about Microsoft Graph API, visit Microsoft Graph Reference.

Setup

The Microsoft Graph data collector works over the Microsoft products, such as Microsoft Azure Directory. To active the alerts and secure score resources from the Microsoft Graph API, a subscription on Microsoft Azure Directory followed by an app registration should be created, as well as configuring the resources with the right permissions for the best performance of the collector.

Setting up permissions on the subscription

Go to the Azure portal and click Azure Activity Directory.
Click App registrations → New registration to create a new app.
Image Removed
On the Register an Application page, give your application a name.
- On Supported Accounts Type, select the third option (Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) )
- On Redirect URI (optional), leave it blank (as default) and click Register.
After registering the app, it will be displayed in a list on the App registration page. Click your app to give it permissions and configure it. You’ll see the app on the dashboard with some important information, docs, and endpoints.
On the left menu, click Authentication → Add a platform → Mobile and desktop applications.
Image Removed
Mark the 3 redirect URIs and click configure.
Image Removed
On the left menu, click API permissions and check if you already have Microsoft Graph on the API/ Permission list. If not, click Add permission and add Microsoft Graph.
Image Removed
Now select Application permissions and search for Security. Check all the boxes available for the service. Then, repeat the same process with Audit and User. If you have done everything correctly, your permissions will display as shown on the green box line. Then Grant admin consent for the applications.
Image RemovedImage RemovedImage Removed

Note

Troubleshooting

If you get this error “Unable to save changes. One or more of the following permission(s) are currently not supported: SecurityEvents.ReadWrite.All, SecurityEvents.Read.All, SecurityActions.Read.All, SecurityActions.ReadWrite.All. Please remove these permission(s) and retry your request. [O6b9]” you might not have set up the permission correctly. Make sure that your configuration is exactly the same as in the green box in the capture above.

Authentication

After applying the permissions, select Certificates & secrets → New client secret, enter the desired name, and copy the token.

Note
The token will display just once. You might have to create another in case you don’t copy it.

...

Getting the credentials

After creating the token, go to Overview to get your Tenant ID and Client ID. This information will be used on the collector server to run the application.

...

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

...

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

...

Code Block

<any_directory>
└── devo-collectors/
    └── msgraph/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        └── config/ 
            └── config-msgraph.yaml

...

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/msgraph/certs. Learn more about security credentials in Devo here.

...

In the config-msgraph.yaml file, replace the <short_unique_identifier>, <tenant_id_value>, <client_id_value>, and <client_secret_value> values and enter the ones that you got in the previous steps. See inline comments to better understand the file.

...

.*.msgraph

API limits, delays and known issues

The Graph API imposes some throttling limits to the applications. The number of requests in a given time is limited according to several criteria. There is a generic limitation for the Graph API and service specific limits.

Any request can be evaluated against multiple limits, depending on the scope of the limit (per app across all tenants, per tenant for all apps, per app per tenant, and so on), the request type (GET, POST, PATCH, and so on), and other factors. The first limit to be reached triggers throttling behavior.

In the case of the services used by the collector, the limits are:

Request type	Per app across all tenants
Any	130,000 requests per 10 seconds
`provisioning_audits`	5 requests per 10 seconds
`directory_audits`	5 requests per 10 seconds
`signins`	5 requests per 10 seconds
`signIns_v2`	5 requests per 10 seconds
`alerts`	150 requests per minute
`alerts_v2`	150 requests per minute
`secure_scores`	150 requests per minute or 10,000 in a 10-minute period
`secure_score_control_proﬁles`	10,000 in a 10-minute period

A complete reference of the limits can be found here.

When the collector reaches the API limit, the API returns a 429 error code. This is reflected in the logs of the collector with a message like 429 Error during API call. If several of these messages are found, the solution is to use the request_period_in_seconds for this service to decrease the frequency of calls.
For instance, a request period of 300 means that the API will be called every 5 minutes for this service, instead of the default value of 60 seconds.

API delays

Microsoft SLAs can be anywhere from 3 minutes to 6 hours in most cases. Check more information here.

Vendor setup

Anchor

	vendor-setup
	vendor-setup

Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:

An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.

You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.

Note
You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits.

Action

Steps

1

Register and configure the application

Go to Azure portal and click on Azure Active Directory.
Click on App registration on the left-menu side. Then click on + New registration.
On the Register and Application page:
1. Name the application.
2. Select Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) in Supported Accounts type.
3. In Redirect URI (optional) leave it as default (blank).
4. Click Register.
App registration page will open. Click on your app to configure it and give it permissions. You will see your app’s dashboard with information (docs, endpoints, etc.) when clicking it.
Click Authentication on the left-menu side, then choose + Add a platform and select Mobile and desktop application.
Select the three redirects URIs:
- https://login.microsoftonline.com/common/oauth2/nativeclient
- https://login.live.com/oauth20_desktop.srf
- msale36f3a02-3eef-437b-874e-8a0aa29a2bf0://auth
Click Configure.

2

Grant the required permissions

Go to API permissions on the left-menu side.
Click + Add permission in case you don’t have Microsoft Graph in the API/Permission list.
Select Application permissions and check SecurityEvents.Read.All.
Check the following permissions: AuditLog.Read.All,Directory.Read.All and User.Read.All. If you did everything correctly, permissions will display.
Select Grant admin consent for the applications.

Info
You do not need to activate permissions if you are not going to use its corresponding resource. Check the Permissions reference per service section for a detailed breakdown on resource and their needed permissions.

3

Obtain the requires credentials for the collector

Go to Certificates & Secrets, select + New client secret . Named it and copy the token value.
Go to Overview to get your Tenant ID and Client ID and copy both values.

Note
The token will display only once. You will need to create another one if you didn’t copy it the first time.

Permission reference per service

Collector service	Resource	Required permissions	Microsoft documentation
`alerts`	`alerts`	`SecurityEvents.Read.All`	List alerts
`alerts_v2`	`alerts_v2`	`SecurityAlert.Read.All, SecurityIncidents.Read.All, and SecurityActions.Read.All`	List alerts_v2
`secure_scores`	`secureScores`	`SecurityEvents.Read.All`	List secureScores
`secure_score_control_profiles`	`secureScoreControlProfiles`	`SecurityEvents.Read.All`	List secureScoreControlProfiles
`directory_audit`	`directoryAudits`	`AuditLog.Read.All and Directory.Read.All`	List directoryAudits
`provisioning_audit`	`provisioningObjectSummary`	`AuditLog.Read.All and Directory.Read.All`	List provisioningObjectSummary
`signIn`	`signIns`	`AuditLog.Read.All and Directory.Read.All`	List signIns
Required for all services	`authentication`	`User.Read`	Microsoft Graph permissions

Note

Troubleshooting

Sometimes you’ll see this error: Unable to save changes. One or more of the following permission(s) are currently not supported: SecurityEvents.Read.All or SecurityActions.Read.All. Please remove these permission(s) and retry your request. [O6b9].

It might that you did not set up the permission correctly. Please, make sure that the permissions are exactly are showing above.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

Info
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

tenant_id_value

This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

client_id_value

This is the Client’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

Note
Sometimes you will find the client_id as Application (client) ID.

client_secret_value

This is the Client’s secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application.

Info
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Accepted authentication methods

This collector only accepts one single authentication method. You will have to fill the following properties on the credentials configuration block:

Authentication method

Tenant ID

Client ID

Client secret

OAuth2 Client credentials

Status

colour	Green
title	REQUIRED

Status

colour	Green
title	REQUIRED

Status

colour	Green
title	REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Use the following command to add the Docker image to the system:

Rw ui tabs macro

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
        # Azure Directory Tenant ID└──       client_id: <client_id_value>               config.yaml

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Image Added

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

globals:
  debug: false
  id: <collector_id_value>
  name: <collector_name_value>
  persistence:
    type: filesystem
    config:
  # Application client ID directory_name: state
outputs:
   client_secret: <client_secret_value>devo_us_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  microsoft_graph_audit:
# Application client secret id:    services:                                                  # Services available for this collector are Alerts, Secure Score and Secure score control profile
      alerts:                                           
        request_period_in_seconds: 60                          # Setting up time interval between API requests. 60sec is recommended.                
#        start_time: "9999-12-31T23:59:59.999Z"                 # Collector Initial time. if this is blank or the data format is not valid, "time now" will be auto filled. 
      secure_scores:
        request_period_in_seconds: 86400                       # 86400 seconds => 1day in seconds, this is the recommended value, but it can be changed
#        start_time: "9999-12-31T23:59:59Z"
      secure_score_control_profile:
        request_period_in_seconds: 86400                       # 86400 seconds => 1day in seconds, this is the recommended value, but it can be changed

Info
If you need to use a custom tag for generated messages, it can be done using the property tag inside any of the available services: alerts, secure_scores, and/or secure_score_control_profile. For example: tag: my.app.msgraph_security.{service_name}

Note

The start_time fields are optional. Follow these formats if you need to use them:

Alerts “start_time” format: 0000-00-00T00:00:000Z

Secure Scores “start_time” format: 0000-00-00T00:00:00Z

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-msgraph-docker-image-1.1.3.tgz

777710bc6194664194796699ea6daade31f20ead5b461894082651e617154dce

Code Block
gunzip -c collector-msgraph-docker-image-<version>.tgz \| docker load

Info
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace "`<version>`" with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/msgraph/

Code Block

docker run \
--name collector-msgraph \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-msgraph.yaml \
--rm -it docker.devo.internal/collector/msgraph:<version>

Note
Replace `<version>` with a proper value.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/msgraph/ directory.

Code Block

version: '3'
services:
  collector-msgraph:
    build:
      context: .
      dockerfile: Dockerfile
    image: docker.devo.internal/collector/msgraph:${IMAGE_VERSION:-latest}
    container_name: collector-msgraph
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-msgraph.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/msgraph/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note

Replace <version> with a proper value.

<short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    environment: <environment_value>
    override_top_level_domain: <override_top_level_domain_value>
    override_base_url: <override_base_url_value>
    override_login_url: <override_login_url_value>
    override_scope_value: <override_scope_value_value>
    services:
      directory_audits:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      provisioning_audits:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      signIns:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      signIns_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
  microsoft_graph_security:
    id: <short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    environment: <environment_value>
    override_top_level_domain: <override_top_level_domain_value>
    override_base_url: <override_base_url_value>
    override_login_url: <override_login_url_value>
    override_scope_value: <override_scope_value_value>
    services:
      secure_score_control_profiles:
        request_period_in_seconds: <request_period_in_seconds_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
      secure_scores:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      alerts:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      alerts_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Replace the placeholders with your required values following the description table below:

Parameter

Data type

Type

Value Range / Format

Details

collector_id_value

str

mandatory

minimum length: 1
maximum length: 5

Use this parameter to give a unique ID to this collector.

collector_name_value

str

mandatory

minimum length: 1
maximum length: 10

Use this parameter to give a name to this collector.

devo_address

str

mandatory

One of:

collector-us.devo.io
collector-eu.devo.io

Use this parameter to identify the Devo Cloud where the events will be sent.

chain_filename

str

mandatory

minimum length: 4
maximum length: 20

Use this parameter to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt.

cert_filename

str

mandatory

minimum length: 4
maximum length: 20

Use this parameter to identify the file.cert downloaded from your Devo domain.

key_filename

str

mandatory

minimum length: 4
maximum length: 20

Use this parameter to identify the file.key downloaded from your Devo domain.

short_unique_id

str

mandatory

minimum length: 5
maximum length: -

Use this parameter to give a unique ID to this input service.
This parameter is used to build the persistence address; do not use the same value for multiple collectors. It could cause a collision.

tenant_id_value

str

mandatory

minimum length: 1

This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

client_id_value

str

mandatory

minimum length: 1

This is the Application (client) ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

client_secret_value

str

mandatory

minimum length: 1

This is the client secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application.

environment_value

str

optional

minimum length: 1

This is an optional control parameter that is injected into the events and allows you to differentiate the environment. For example: dev and prod.

Info
This parameter should be removed if it is not used.

request_period_in_seconds_value

int

optional

minimum length: 60

Period in seconds used between each data pulling. This value will overwrite the default value (300 seconds).

start_time_in_utc_value

str

optional

UTC datetime string having datetime string format %Y-%m-%dT%H-%M-%SZ (e.g., “2020-01-01T00:00:01Z”)

This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events.

Info
This parameter should be removed if it is not used.

additional_filter_value

str

optional

minimum length: 1

This parameter allows you to add an additional Microsoft Graph OData query clause to the default filters. By default, the collector attempts to fetch all data possible for a given Microsoft Graph entity type.

See OData query parameters documentation for more information on how to format filters for Microsoft Graph API.

Info
This parameter should be removed if it is not used.

override_tag_value

str

optional

Devo tag-friendly string (no special characters, spaces, etc.) For more information see Devo Tags

An optional tag that allows users to override the service default tags.

Info
This parameter should be removed if it is not used.

override_top_level_domain_value

str

optional

One of:

com
us

Default value:com

This is an optional parameter that allows you to override the default top level domain.

If you're working in a Microsoft 365 GCC environment, continue using the worldwide endpoints (com).

If you're working in a Microsoft 365 GCC High environment, use us.

Info
This parameter should be removed if it is not used.

override_base_url_value

str

optional

default value: https://graph.microsoft.{top_level_domain}

A parameter that allows you to override the base URL.

Info
This parameter should be removed if it is not used.

override_login_url_value

str

optional

default value: https://login.microsoftonline.com/

A parameter that allows you to override the login URL.

Info
This parameter should be removed if it is not used.

override_scope_value

str

optional

default value: https://graph.microsoft.com/.default

A parameter that allows you to override the scope value.

Info
This parameter should be removed if it is not used.

override_query_window_max_seconds

int

optional

default value: None

A parameter that allows you to override the maximum query window. This is useful for any APIs that greatly restrict the amount of data one can fetch for a given datetime range (performance reasons). The collector will continuously fetch data but in smaller increments.

Info
This parameter should be removed if it is not used.

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-ms-graph-collector-if-docker-image-2.0.1	`0dcd32b48f769fd597f61edb76f83086f8fcfc16dc8003b9e6a68388a031ddd3`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <collector-ms-graph-collector-if-docker-image>-<1.2.0>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

In the Collector Server GUI, access the domain in which you want this instance to be created
Click Add Collector and find the one you wish to add.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.
In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

Code Block

{
  "global_overrides": {
    "debug": false
  },
  "inputs": {
    "microsoft_graph_audit": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "tenant_id": "<tenant_id_value>",
        "client_id": "<client_id_value>",
        "client_secret": "<client_secret_value>"
      },
      "environment": "<environment_value>",
      "override_top_level_domain": "<override_top_level_domain_value>",
      "override_base_url": "<override_base_url_value>",
      "override_login_url": "<override_login_url_value>",
      "override_scope_value": "<override_scope_value_value>",
      "services": {
        "directory_audits": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "provisioning_audits": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "signIns": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "signIns_v2": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        }
      }
    },
    "microsoft_graph_security": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "tenant_id": "<tenant_id_value>",
        "client_id": "<client_id_value>",
        "client_secret": "<client_secret_value>"
      },
      "environment": "<environment_value>",
      "override_top_level_domain": "<override_top_level_domain_value>",
      "override_base_url": "<override_base_url_value>",
      "override_login_url": "<override_login_url_value>",
      "override_scope_value": "<override_scope_value_value>",
      "services": {
        "secure_score_control_profiles": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>"
        },
        "secure_scores": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "alerts": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "alerts_v2": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        }
      }
    }
  }
}

Parameter

Data type

Type

Value Range / Format

Details

short_unique_id

str

mandatory

minimum length: 1
maximum length: 5

Use this parameter to give a unique ID to this input service.
This parameter is used to build the persistence address; do not use the same value for multiple collectors. It could cause a collision.

tenant_id_value

str

mandatory

minimum length: 1

This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

client_id_value

str

mandatory

minimum length: 1

This is the Application (client) ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

client_secret_value

str

mandatory

minimum length: 1

This is the client secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application.

environment_value

str

optional

minimum length: 1

This is an optional control parameter that is injected into the events and allows you to differentiate the environment. For example: dev and prod.

Info
This parameter should be removed if it is not used.

request_period_in_seconds_value

int

optional

minimum length: 60

Period in seconds used between each data pulling. This value will overwrite the default value (300 seconds).

start_time_in_utc_value

str

optional

UTC datetime string having datetime string format %Y-%m-%dT%H-%M-%SZ (e.g., “2020-01-01T00:00:01Z”)

This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events.

Info
This parameter should be removed if it is not used.

additional_filter_value

str

optional

minimum length: 1

This parameter allows you to add an additional Microsoft Graph OData query clause to the default filters. By default, the collector attempts to fetch all data possible for a given Microsoft Graph entity type.

See OData query parameters documentation for more information on how to format filters for Microsoft Graph API.

Info
This parameter should be removed if it is not used.

override_tag_value

str

optional

Devo tag-friendly string (no special characters, spaces, etc.) For more information see Devo Tags

An optional tag that allows users to override the service default tags.

Info
This parameter should be removed if it is not used.

override_top_level_domain_value

str

optional

One of:

com
us

Default value:com

This is an optional parameter that allows you to override the default top level domain.

If you're working in a Microsoft 365 GCC environment, continue using the worldwide endpoints (com).

If you're working in a Microsoft 365 GCC High environment, use us.

Info
This parameter should be removed if it is not used.

override_base_url_value

str

optional

default value: https://graph.microsoft.{top_level_domain}

A parameter that allows you to override the base URL.

Info
This parameter should be removed if it is not used.

override_login_url_value

str

optional

default value: https://login.microsoftonline.com/

A parameter that allows you to override the login URL.

Info
This parameter should be removed if it is not used.

override_scope_value

str

optional

default value: https://graph.microsoft.com/.default

A parameter that allows you to override the scope value.

Info
This parameter should be removed if it is not used.

override_query_window_max_seconds

int

optional

default value: None

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Expand

title	directory_audits

Internal process and deduplication method

All directory audit records are continuously pulled subject to the activityDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

All events of this service are ingested into the table cloud.azure.ad.audit.

Expand

title	provisioning_audits

Internal process and deduplication method

All provisioning audit records are continuously pulled subject to the activityDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

All events of this service are ingested into the table cloud.azure.ad.provisioning.

Expand

title	signIns

Internal process and deduplication method

All sign-in records are continuously pulled subject to the createdDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

All events of this service are ingested into the table cloud.azure.ad.signin.

Expand

title	signIns_v2

Internal process and deduplication method

All sign-in_v2 records are continuously pulled subject to the createdDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

Events of this service are ingested into the following tables:

signIn event type	Devo table
interactiveUser	`cloud.azure.ad.interactive_user_signin`
nonInteractiveUser	`cloud.azure.ad.noninteractive_user_signin`
managedIdentity	`cloud.azure.ad.managed_identity_signin`
servicePrincipal	`cloud.azure.ad.service_principal_signin`
unknownFutureValue	`cloud.azure.ad.unknown_future_value_signin`

Expand

title	alert

Internal process and deduplication method

All alert records are continuously pulled subject to the createdDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

Events of this service are ingested into the following tables:

Vendor	Devo table
IPC	`cloud.azure.ad.alerts`
MCAS	`cloud.office365.cloud_apps.alerts`
Microsoft Defender ATP	`cloud.office365.endpoint.alerts`
Microsoft 365 Defender	`cloud.office365.endpoint.alerts`
Office 365 Security and Compliance	`cloud.office365.security.alerts`
Azure Sentinel	`cloud.azure.sentinel.alerts`
ASC	`cloud.office365.identity.alerts`
Azure Advanced Threat Protection	`cloud.azure.securitycenter.alerts`

Expand

title	alerts_v2

Internal process and deduplication method

All alerts_v2 records are continuously pulled subject to the activityDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

All events of this service are ingested into the table cloud.msgraph.security.alerts_v2.

Expand

title	secure_scores

Internal process and deduplication method

All secure score records are continuously pulled subject to the createdDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

All events of this service are ingested into the table cloud.office365.security.scores.

Expand

title	secure_score_control_profiles

Internal process and deduplication method

All secure score profile records are continuously pulled subject to the createdDateTime timestamp property. A unique hash value is computed for each event and used for deduplication purposes to ensure events are not fetched multiple times in subsequent pulls. The collector utilizes the @odata.nextLink link (if provided) to fetch the next page of records for a given query window until no more remain.

Devo categorization and destination

All events of this service are ingested into the table cloud.office365.security.scores.

Restart the persistence for a service

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the start_time_in_utc parameter to a different one.
Save the changes.
Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Common logic

Error type	Error ID	Error message	Cause	Solution
`InitVariablesError`	1	Invalid `start_time_in_utc: {ini_start_str}`. Must be in parseable datetime format.	The configured `start_time_in_utc` parameter is a non-parseable format.	Update the `start_time_in_utc` value to have the recommended format as indicated in the guide.
`InitVariablesError`	2	Invalid `start_time_in_utc: {ini_start_str}`. Must be in the past.	The configured `start_time_in_utc` parameter is a future date.	Update the `start_time_in_utc` value to a past datetime.
`SetupError`	101	Failed to fetch OAuth token from `{token_endpoint}`. Exception: `{e}`.	The provided credentials, base URL, and/or token endpoint are incorrect.	Revisit the configuration steps and ensure that the correct values were specified in the config file.
`SetupError`	102	Failed to fetch data from `{endpoint}`. Source is not pullable.	The provided credentials, base URL, and/or token endpoint are incorrect.	Revisit the configuration steps and ensure that the correct values were specified in the config file.
`ApiError`	401	Error during API call to [API provider HTML error response here]	The server returned an HTTP 401 response.	Ensure that the provided credentials are correct and provide read access to the targeted data.
`ApiError`	429	Error during API call to [API provider HTML error response here]	The server returned an HTTP 429 response.	The collector will attempt to retry requests (default up to 3 times) and respect back-off headers if they exist. If the collector repeatedly encounters this error, adjust the `request_period_in_seconds`. (see section about API limits)
`ApiError`	498	Error during API call to [API provider HTML error response here]	The server returned an HTTP 500 response.	If the API returns a 500 but successfully completes subsequent runs then you may ignore this error. If the API repeatedly returns a 500 error, ensure the server is reachable and operational.

Collector operations

Verify collector operations

Expand

title	Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module:

Code Block

2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-test-local.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> "\etc\devo\job" does not exists
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> "\etc\devo\collector" does not exists
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "C:\git\collectors2\devo-collector-<name>\config\config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
2023-01-10T15:22:57.171 WARNING MainProcess::MainThread -> [WARNING] Illegal global setting has been ignored -> multiprocessing: False

Expand

title	Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Code Block

2023-01-10T15:23:00.788 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting
thread
2023-01-10T15:23:00.789 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) ->
Starting thread (every 300 seconds)
2023-01-10T15:23:00.790 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) ->
Starting thread
2023-01-10T15:23:00.842 INFO OutputProcess::MainThread -> global_status: {"output_process": {"process_id": 18804,
"process_status": "running", "thread_counter": 21, "thread_names": ["MainThread", "pydevd.Writer", "pydevd.Reader",
"pydevd.CommandThread", "pydevd.CheckAliveThread", "DevoSender(standard_senders,devo_sender_0)",
"DevoSenderManagerMonitor(standard_senders,devo_1)", "DevoSenderManager(standard_senders,manager,devo_1)",
"OutputStandardConsumer(standard_senders_consumer_0)",

Expand

title	Sender services

The Integrations Factory Collector SDK has 3 different sender services depending on the event type to deliver (internal, standard, and lookup). This collector uses the following Sender Services:

Logging trace	Description
Number of available senders: 1	Displays the number of concurrent senders available for the given Sender Service.
Sender manager internal queue size: 0	Displays the items available in the internal sender queue. This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.
Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)	Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained: 44 events were sent to Devo since the collector started. The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00. 21 events were sent to Devo between the last UTC checkpoint and now. Those 21 events required 0.007 seconds to be delivered. By default these traces will be shown every 10 minutes.

Expand

title	Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace	Description
Number of available senders: 1	Displays the number of concurrent senders available for the given Sender Service
Sender manager internal queue size: 0	Displays the items available in the internal sender queue.
Standard - Total number of messages sent: 57, messages sent since "2023-01-10 16:09:16.116750+00:00": 0 (elapsed 0.000 seconds	Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained: 44 events were sent to Devo since the collector started. The last checkpoint timestamp was 2023-01-10 16:09:16.116750+00:00. 21 events were sent to Devo between the last UTC checkpoint and now. Those 21 events required 0.00 seconds to be delivered.

Check memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

Code Block

INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB ->
 410.02MiB)
 INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB ->
 705.28MiB)Change log

Anchor

	section
	section

1.X to 2.X migration guide

Version 2.0.0 introduced several changes to the collector's configuration. When upgrading the collector, users must make changes to their configuration file to ensure that the collector continues to work as expected.

Note

Important

Note that tag_version behaviour has been replaced by override_tag

In versions 1.X of the collector, some services had a config parameter tag_version with values v1or v2. The effect of this parameter is that the destination table for these services will be different. These are the destination tables according to the tag_version value:

v1	v2 (default)
`cloud.msgraph.security.score`	`cloud.office365.security.score`
`cloud.msgraph.security.scorecontrol`	`cloud.office365.security.scorecontrol`

In the new collector 2.0.0, the old config parameter tag_version has been removed. The same effect as v1can be made using override_tag, with these values:

Service	`override_tag` (to obtain the behaviour equivalent to `tag_version: v1`)
`secure_scores`	`cloud.msgraph.security.scores.2`
`secure_score_control_profiles`	`cloud.msgraph.security.scorecontrol.2`

For instance, if we have this in the old config:

Code Block
secure_scores: tag_version: v1 secure_score_control_profile: tag_version: v1

you should change it to:

Code Block
secure_scores: override_tag: cloud.msgraph.security.scores.2 secure_score_control_profile: override_tag: cloud.msgraph.security.scorecontrol.2

Note

Important

Note that security alerts are sent to different tables according to their categories.

In old versions of this collector, all alerts were sent to the table cloud.msgraph.security.alerts

However, as there are alerts of several types, in version 2.0.0 alerts are categorized according to their vendorInformation-provider field (their vendor) and sent to different tables:

Vendor	Old table	New table
IPC	`cloud.msgraph.security.alerts`	`cloud.azure.ad.alerts`
MCAS		`cloud.office365.cloud_apps.alerts`
Microsoft Defender ATP		`cloud.office365.endpoint.alerts`
Microsoft 365 Defender		`cloud.office365.endpoint.alerts`
Office 365 Security and Compliance		`cloud.office365.security.alerts`
Azure Sentinel		`cloud.azure.sentinel.alerts`
ASC		`cloud.office365.identity.alerts`
Azure Advanced Threat Protection		`cloud.azure.securitycenter.alerts`
Others		`cloud.msgraph.security.alerts`

It is possible to avoid this feature and send all alerts to the same table by editing the config file and changing the tag:

Code Block
alerts: override_tag: cloud.msgraph.security.alerts.2

Config file comparison

Below you will find a sample 1.7.1 configuration file and a sample 2.0.0 configuration file.

Expand

title	1.7.1 Configuration (inputs)

Code Block

inputs:
  microsoft_graph:
    id: <short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    services:
      audit:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      provisioning:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn_nonInteractive:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn_servicePrincipal:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn_managedIdentity:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      secure_score_control_profile:
        tag_version: v2
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      alerts:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        pull_sliding_window_timespan_period: <pull_sliding_window_timespan_period_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      secure_scores:
        tag_version: v2
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        pull_sliding_window_timespan_period: <pull_sliding_window_timespan_period_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      alerts_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>

Expand

title	2.0.0 Configuration (inputs)

Code Block

inputs:
  microsoft_graph_audit:
    id: <short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    environment: <environment_value>
    override_top_level_domain: <override_top_level_domain_value>
    override_base_url: <override_base_url_value>
    override_login_url: <override_login_url_value>
    override_scope_value: <override_scope_value_value>
    services:
      directory_audits:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      provisioning_audits:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      signIns:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      signIns_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
  microsoft_graph_security:
    id: <short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    environment: <environment_value>
    override_top_level_domain: <override_top_level_domain_value>
    override_base_url: <override_base_url_value>
    override_login_url: <override_login_url_value>
    override_scope_value: <override_scope_value_value>
    services:
      secure_score_control_profiles:
        request_period_in_seconds: <request_period_in_seconds_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
      secure_scores:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      alerts:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      alerts_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>

Config file changes

The URL endpoints (override_base_url_main, override_base_url_vendor, override_base_url_vendor_with_sub_provider , override_login_url) have been moved from the individual services to the global configuration section.
override_base_url_main has been renamed to override_base_url.
tag_version has been removed.
pull_sliding_window_timespan_period has been removed.
reset_persistence_auth has been removed.
override_time_delta_in_days has been removed.
ms_365_environment has been replaced by override_top_level_domain. GCC High Gov should use us in the override_top_level_domain. Alternatively, users can use override_base_url to specify the GCC High Gov base URL.
additional_filter has been added to all services. Users can use this field to specify additional filters that will be applied when querying the Microsoft Graph API.
The collector can use new services from Graph (beta endpoint in Graph), that services, that use to be in a separate service for each type, have been consolidated into one service called signIns_v2. Users should remove all three services from their config and use only the signIns_v2 service.
start_time has been renamed to start_time_in_utc.

Persistence changes

The persistence object consists of the following fields: persistence_version, last_event_time_in_utc, last_ids, and next_link.
The collector will automatically map old key names (e.g. last_polled_timestamp → last_event_time_in_utc) to the appropriate value.

Change log

Release

Released on

Release type

Details

Recommendations

v2.0.1

02 Aug 2024

Status

colour	Green
title	IMPROVEMENT

Status

colour	Red
title	BUG FIX

Improvements

Updated DCSDK to 1.12.2 from 1.11.1

Bug Fixing

authlib updated to 1.3.1 to solve CVE-2024-37568

Recommended version

v2.0.0

01 Apr 2024

Status

colour	Green
title	IMPROVEMENT

Improvements

Complete reimplementation of the collector, refactoring all the services

Upgrade

v1.7.1

18 Oct 2023

Status

colour	Purple
title	NEW FEATURE

Status

colour	Green
title	IMPROVEMENT

Status

colour	Red
title	BUG FIX

New features

time_delta_in_days can now be overwritten in the user configuration by override_time_delta_in_days in all the service that are “time-based”. This new parameter cannot be combined with reset_persistence_auth and/or start_time.

Improvements

The state is now persisted more frequently for most of the services. This means that, in case of a collector restart, the chances of duplicating data have been reduced considerably, as the collector will continue pulling data from the same point where it was when the collector was stopped.

Bug fixing

The collector will get the most recent token available before performing any new request, reducing the possibilities to get a 401 code as a response.
The 504 code responses were returned many time; some of them for having asked for too old data. This used to cause a locking state in the collector, as it was not able to continue. Some mechanisms has been added to avoid requiring that old data to the API. Anyway, if a 504 appears now for any other reason, the improvement related to persisting the state frequently makes the collector continue collecting correctly after the service restart.

Upgrade

v1.7.0

10 Oct 2023

Status

colour	Purple
title	NEW FEATURE

Status

colour	Green
title	IMPROVEMENT

Status

colour	Red
title	BUG FIX

New features:

alerts_v2 service included, keeping old alerts service for compatibility.
Compliance for MS 365 GCC High US environments added.

Improvements:

Update DCSDK from 1.8.0 to 1.9.2

Bug fixing:

The collector now keeps retrieving events when it is up-to-date.
Added extra protection to refresh token and avoid 401 status errors.
When a 401 status code is received from a response, the collector tries the request again using the access_token available in the collector_variables, instead of raising en Error. This definitely fixes the bug that used to make the collector restart due to 401 errors.
A vendor thread termination event has been set, including three different check points in the thread's run method, as a protection against non-terminated vendor threads, causing the alerts service to stop. Some extra logging has also been added to identify the root cause in case this keeps happening.

Upgrade

v1.6.2

14 Jan 2023

Status

colour	Red
title	BUG FIX

Status

colour	Green
title	IMPROVEMENT

Improvements:

Update DCSDK from 1.6.0 to 1.8.0

Bug fixing:

Fix in the service urls: they were not being formatted correctly with the start_time variable, which allows the user to select the date from which they want to collect events.
Updated the limits of the API: The limits have been modified with the official values. This fixes throttling issues.
Updated the default value of star_time from 61 days in the past to 30, as this is the maximum limit the API allows.

Upgrade

v1.6.0

02 Aug 2022

Status

colour	Purple
title	NEW FEATURE

New features:

The pulling mechanism now uses a sliding window to avoid event loss and duplication.

Improvements:

DevoCollectorSDK upgraded to v1.6.0:
- Added:
  - More log traces related to execution environment details.
  - Global rate limiters functionality.
  - Extra checks for supporting MacOS as development environment.
  - Obfuscation functionality.
- Changed:
  - Some log traces now are shown less frequently.
  - The default value for the logging frequency for "main" processes has been changed (to 120 seconds).
  - Updated some Python Packages.
  - Controlled stopping functionality more stable when using the "template".
  - Improved some log messages related to Devo certificates (when using the Devo sender).
  - Validate json objects before saving them to persistence (using filesystem).

Upgrade

v1.4.2

27 Dec 2022

Status

colour	Red
title	BUG FIX

Fixed bugs:

Fixes bug with non-time-based puller state.

Upgrade

v1.4.1

02 Dec 2022

Status

colour	Red
title	BUG FIX

Fixed bugs:

Fix error with vendor state when checking the reset_persistence_auth parameter.
Allow using v2 tags for secure_scores and secure_scores_control_profile tags.
Add missing Devo metadata into events.

Upgrade

v1.4.0

02 Dec 2022

Status

colour	Green
title	IMPROVEMENT

Improvements:

Automatic outdated start_time correction for audit-based services.
- New “reset persistence” functionality.

Upgrade

v1.3.0

18 Nov 2022

Status

colour	Green
title	IMPROVEMENT

Status

colour	Red
title	BUG FIX

Improvements:

start_time configuration parameter normalization for audit and provisioning services.
Upgraded devocollectorsdk from 1.4.0 to 1.4.4b:
- Added:
  - New "templates" functionality.
  - New controlled stopping condition when any input thread fatally fails.
  - Log traces for knowing the execution environment status (debug mode).
- Changed:
  - Improved log trace details when runtime exceptions happen
  - Refactored source code structure
  - Fixes in the current puller template version
  - The Docker container exits with the proper error code

Bug fixing:

Correct token validation when a Partial Content response is received.
Use appropriate destination tag for provisioning events.

Upgrade

v1.2.0

02 Aug 2022

Status

colour	Purple
title	NEW FEATURE

Status

colour	Green
title	IMPROVEMENT

New features:

New supported sources
- Sign In (signIn service)
- Audit (audit service)
- Provisioning (provisioning service)
Previous services modification
- The new tagging introduced in the previous v1.1.3 release is now customizable through the tag_version service parameter. The default tagging has been reverted to the original one.
- The alerts source, when setting the tag_version to v2, will try to categorize the events by applying different tags based on the event’s provider.

Improvements:

Token validation is now performed against the corresponding endpoint.

Upgrade

Page Comparison

Versions Compared

Old Version 2

New Version Current

Key

Service description

Overview

Configuration requirements

Devo collector features

Data sources

Data source description

Microsoft Graph resources

Setup

Setting up permissions on the subscription

Authentication

Getting the credentials

Run the collector

API limits, delays and known issues

API delays

Vendor setup

Permission reference per service

Minimum configuration required for basic pulling

Accepted authentication methods

Run the collector

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Download the Docker image

Docker

Docker Compose

Editing the JSON configuration

Collector services detail

Internal process and deduplication method

Devo categorization and destination

Internal process and deduplication method

Devo categorization and destination

Internal process and deduplication method

Devo categorization and destination

Internal process and deduplication method

Devo categorization and destination

Internal process and deduplication method

Devo categorization and destination

Internal process and deduplication method

Devo categorization and destination

Internal process and deduplication method

Devo categorization and destination

Internal process and deduplication method

Devo categorization and destination

Restart the persistence for a service

Troubleshooting

Common logic

Collector operations

Verify collector operations

Check memory usage

1.X to 2.X migration guide

Config file comparison

Config file changes

Persistence changes

Change log