Microsoft Graph collector

Microsoft Graph collector

Configuration requirements

To run this collector, there are some configurations detailed below that you need to consider.

Configuration

Details

Configuration

Details

Azure account

Azure account with admin level permissions and Azure AD tenant.

Credentials

The credentials configuration block has been filled correctly.

More information

Refer to the Vendor setup section to know more about these configurations.

Overview

The Microsoft Graph Collector provides the ability to collect data and intelligence from services such as Microsoft 365, Windows, and Enterprise Mobility and Security. This data collector is able to ingest security alerts, scores, provisioning, audit, and sign-ins retrieved from Microsoft products, allowing you to empower streamlined security operations and better defend against threats faced in Azure AD and Microsoft 365 environments.

Devo collector features

Feature

Details

Feature

Details

Allow parallel downloading (multipod)

  • Allowed

Running environments

  • Collector server

  • On-premise

Populated Devo events

  • Table

Flattening preprocessing

  • No

Data sources

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Alerts

Represents potential security issues within a customer’s tenant that Microsoft or partner security solutions have identified.

Refer to Microsoft documentation about Alert Resource Type for more information.

https://graph.microsoft.com/v1.0/security/alerts?$count=true&$filter=eventDateTime+ge+{start_time}+AND+vendorInformation/provider+eq+'{provider}'&$orderby=eventDateTime+asc&$top={items_per_vendor_request}

alerts

Starting from v1.2.0, the destination table depends on the tag_version configuration parameter:

  • v1: ºv2: the destination will be dynamic, depending on the provider:

    • IPC: cloud.azure.ad.alerts.

    • MCAS: cloud.office365.cloud_apps.alerts.

    • Microsoft Defender ATP: cloud.office365.endpoint.alerts.

    • Office 365 Security and Compliance: cloud.office365.security.alerts.

    • Azure Sentinel: cloud.azure.sentinel.alerts.

    • ASC: cloud.office365.identity.alerts.

    • Azure Advanced Threat Protection: cloud.azure.securitycenter.alerts.

    • For any other provider, it will fall back in v1's tagging.

The detailed table destination will depend on the tag_version’s value v1.0.0available in the Devo categorization and destination section.

v1.0.0

Secure scores

Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held.

Refer to the Microsoft documentation for more information about Secure scores resources types.

https://graph.microsoft.com/v1.0/security/secureScores?$count=true&$filter=createdDateTime+ge+{start_time}+AND+vendorInformation/provider+eq+'{provider}'&$orderby=createdDateTime+asc,vendorInformation/provider+asc&$top={items_per_vendor_request}

secure_scores

Starting from v1.2.0, the destination table depends on the tag_version configuration parameter:

  • v1: cloud.msgraph.security.secure_scores

  • v2: cloud.office365.security.scores

The detailed table destination depending on the tag_version's value is available in the Devo categorization and destination section.

v1.0.0

Secure score control profiles

Represents a tenant's secure score per control data.

Refer to the Microsoft documentation for more information about Secure score control profiles.

https://graph.microsoft.com/v1.0/security/secureScoreControlProfiles?$count=true

secure_score_control_profiles

Starting from v1.2.0, the destination table depends on the tag_version configuration parameter:

  • v1: cloud.msgraph.security.secure_score_control_profiles

  • v2: cloud.office365.security.scorecontrol

The detailed table destination depending on the tag_version's value is available in the Devo categorization and destination section.

v1.0.0

Directory audit

Represents the directory audit items and its collection.

Refer to the Microsoft documentation for more information about Directory audit.

https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?$filter=activityDateTime ge {start_time}&$orderby=activityDateTime+asc&$top={items_per_main_request}

audit

cloud.azure.ad.audit

v1.2.0

Provisioning

Represents an action performed by the Azure AD Provisioning service and its associated properties.

Refer to the Microsoft documentation for more information about Provisioning.

 

https://graph.microsoft.com/beta/auditLogs/provisioning?$filter=activityDateTime ge {start_time}&$orderby=activityDateTime+asc&$top={items_per_main_request}

provisioning

cloud.azure.ad.audit

v1.2.0

Sign-in

Details user and application sign-in activity for a tenant (directory).

Refer to the Microsoft documentation for more information about Sign-in.

 

  • signIn: https://graph.microsoft.com/v1.0/auditLogs/signIns?$orderby=createdDateTime+asc&$top={items_per_main_request}

  • signIn_nonInteractive: https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'nonInteractiveUser')&$orderby=createdDateTime+asc&$top={items_per_main_request}

  • signIn_servicePrincipal: https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'servicePrincipal')&$orderby=createdDateTime+asc&$top={items_per_main_request}

  • signIn_managedIdentity: https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'managedIdentity')&$orderby=createdDateTime+asc&$top={items_per_main_request}

  • signIn

  • signIn_nonInteractive

  • signIn_servicePrincipal

  • signIn_managedIdentity

  • signIn: cloud.azure.ad.signin

  • signIn_nonInteractive: cloud.azure.ad.noninteractive_user_signin

  • signIn_servicePrincipal: cloud.azure.ad.service_principal_signin

  • signIn_managedIdentity: cloud.azure.ad.managed_identity_signin

v1.2.0

Vendor setup

Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:

  1. An Azure account that has an active subscription.

  2. The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).

  3. A working Azure AD tenant.

You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.

You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits.

 

Action

Steps

1

Register and configure the application

  1. Go to Azure portal and click on Azure Active Directory.

  2. Click on App registration on the left-menu side. Then click on + New registration.

  3. On the Register and Application page:

    1. Name the application.

    2. Select Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) in Supported Accounts type.

    3. In Redirect URI (optional) leave it as default (blank).

    4. Click Register.

  4. App registration page will open. Click on your app to configure it and give it permissions. You will see your app’s dashboard with information (docs, endpoints, etc.) when clicking it.

  5. Click Authentication on the left-menu side, then choose + Add a platform and select Mobile and desktop application.

  6. Select the three redirects URIs:

    • https://login.microsoftonline.com/common/oauth2/nativeclient

    • https://login.live.com/oauth20_desktop.srf

    • msale36f3a02-3eef-437b-874e-8a0aa29a2bf0://auth

  7. Click Configure.

2

Grant the required permissions

  1. Go to API permissions on the left-menu side.

  2. Click + Add permission in case you don’t have Microsoft Graph in the API/Permission list.

  3. Select Application permissions and search for Security. Check SecurityEvents.Read.All.

  4. Repeat the same step 3 for AuditLog.Read.All, Directory.Read.All and User.Read. If you did everything correctly, permissions will display.

  5. Select Grant admin consent for the applications.

You do not need to activate permissions if you are not going to use its corresponding resource. Check the Permissions reference per service section for a detailed breakdown on resource and their needed permissions.

3

Obtain the requires credentials for the collector

  1. Go to Certificates & Secrets, select + New client secret . Named it and copy the token value.

  2. Go to Overview to get your Tenant ID and Client ID and copy both values.

The token will display only once. You will need to create another one if you didn’t copy it the first time.

Permission reference per service

Collector service

Resource

Required permissions

Microsoft documentation

alerts

alerts

SecurityEvents.Read.All

List alerts

secure_scores

secureScores

SecurityEvents.Read.All

List secureScores

secure_score_control_profiles

secureScoreControlProfiles

SecurityEvents.Read.All

List secureScoreControlProfiles

audit

directoryAudits

AuditLog.Read.All and Directory.Read.All

List directoryAudits

provisioning

provisioningObjectSummary

 

AuditLog.Read.All and Directory.Read.All

List provisioningObjectSummary

signIn

signIns

AuditLog.Read.All and Directory.Read.All

List signIns

Required for all services

authentication

User.Read

Microsoft Graph permissions

Troubleshooting

Sometimes you’ll see this error: Unable to save changes. One or more of the following permission(s) are currently not supported: SecurityEvents.Read.All or SecurityActions.Read.All. Please remove these permission(s) and retry your request. [O6b9].

It might that you did not set up the permission correctly. Please, make sure that the permissions are exactly are showing above.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

tenant_id_value

This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

client_id_value

This is the Client’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.

Sometimes you will find the client_id as Application (client) ID.

client_secret_value

This is the Client’s secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application.

See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Accepted authentication methods

This collector only accepts one single authentication method. You will have to fill the following properties on the credentials configuration block:

Authentication method

Tenant ID

Client ID

Client secret

OAuth2 Client credentials

REQUIRED

REQUIRED

REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Alerts

Alerts are potential security issues within a customer's tenant that Microsoft or partner security solutions have identified and flagged for action or notification. With the Microsoft Graph alerts entity, you can unify and streamline the management of security issues across all integrated solutions.

Alerts Security Providers:

  • Microsoft Defender for Cloud

  • Azure Active Directory Identity Protection

  • Microsoft Defender for Cloud Apps

  • Microsoft Defender for Endpoint

  • Microsoft Defender for Identity

  • Microsoft 365

  • Azure Information Protection

  • Azure Sentinel

Secure Scores

Microsoft Secure Score is a security analytics solution that gives you visibility into your security portfolio and how to improve it. With a single score, you can better understand what you have done to reduce your risk in Microsoft solutions. You can also compare your score with other organizations and see how your score has been trending over time. The Microsoft Graph secureScore and secureScoreControlProfile entities help you balance your organization's security and productivity needs while enabling the appropriate mix of security features. You can also project what your score would be after you adopt security features.

Azure Active Directory reports (Sign-in, Audit, Provisioning)

Azure Active Directory (Azure AD) reports providing a comprehensive view of activity in your environment. The provided data enables you to:

  • Determine how your apps and services are utilized by your users.

  • Detect potential risks affecting the health of your environment.

  • Troubleshoot issues preventing your users from getting their work done.

These reports help you understand the behavior of users in your organization. There are three types of reports that this collector pulls from Azure AD:

  • Sign-ins: Information about sign-ins and how your resources are used by your users.

  • Audit: Information about changes applied to your tenants such as users and group management or updates applied to your tenant’s resources.

  • Provisioning: Activities performed by the provisioning service, such as the creation of a group in ServiceNow or a user imported from Workday.

Devo categorization and destination

Here you can see how each service that has configurable tagging will tag its events depending on the value of the tag_version parameter:

Service

Old tagging (v1)

New tagging (v2)

secure_scores

cloud.msgraph.security.secure_scores.1

cloud.office365.security.scores.1.msgraph

secure_score_control_profiles

cloud.msgraph.security.secure_score_control_profiles.1

cloud.office365.security.scorecontrol.1.msgraph

alerts

cloud.msgraph.security.alerts.1

The tagging is based on the provider. See the next table.

The alerts Service Now uses dynamic tagging based on the event’s provider field. This is the provider/tag correspondence:

Provider

New tagging (v2)

IPC

cloud.azure.ad.alerts.1.msgraph

MCAS

cloud.office365.cloud_apps.alerts.1.msgraph

Microsoft Defender ATP

cloud.office365.security.alerts.1.msgraph

Office 365 Security and Compliance

cloud.azure.sentinel.alerts.1.msgraph

Azure Sentinel

cloud.office365.identity.alerts.1.msgraph

Azure Advanced Threat Protection

cloud.azure.securitycenter.alerts.1.msgraph

If the event comes with a provider that is not present in the table above, the tagging will fall back to the old one (v1).

Events service

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Access Token has been validated successfully INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Setup for module <MicrosoftGraphNonTimeBasedPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Starting data collection every 60 seconds INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 2 requests, retrieved all vendor list, detected 1 unique vendors INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> MicrosoftGraphNonTimeBasedVendorPuller(microsoft_graph#1,secure_score_control_profile#predefined,MicrosoftGraphNonTimeBasedPuller#SecureScore.None) -> Starting thread INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Data collection completed. Elapsed time: 4.708 seconds. Waiting for 55.292 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

his collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can only do that by re-creating a new collector instance from scratch since this collector does not implement a state restart mechanism.

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Common for all services

Error type

Error ID

Error message

Cause

Solution

Error type

Error ID

Error message

Cause

Solution

MicrosoftGraphPullerCredentialsException

1

Invalid tenant. This may happen if there are no active subscriptions for the tenant

The tenant is not valid according to Microsoft AD. Probably, the setup has not been correctly followed.

Revisit the setup instructions and check that all the steps have been correctly followed.

1

Access Token validation has been failed, code: {status_code}, text: {text_str}

There was a problem obtaining the authentication token.

Read the exact error to understand what is the real cause.

2

Access Token not valid or client_id does not exist

There was an unknown problem during the authentication..

Check that the credentials have been correctly set up.

MicrosoftGraphPullerCreationException
ModuleDefinitionError

1

"module_properties" property in service definition must exists / "module_properties" mandatory property is missing or empty

This a programming error that we should not except to happen.

Contact Devo Support.

2

"module_properties" property in service definition must be a dictionary

This a programming error that we should not except to happen.

Contact Devo Support.

3

"resource_type" property in service definition must exists

This a programming error that we should not except to happen.

Contact Devo Support.

4

"resource_type" property in service definition must be a string

This a programming error that we should not except to happen.

Contact Devo Support.

5

"base_url_main" property in service definition must exists

This a programming error that we should not except to happen.

Contact Devo Support.

6

"base_url_main" property in service definition must be a string.

This a programming error that we should not except to happen.

Contact Devo Support.

7

"base_url_vendor" property in service definition must exists

This a programming error that we should not except to happen.

Contact Devo Support.

8

"base_url_vendor" property in service definition must be a string

This a programming error that we should not except to happen.

Contact Devo Support.

9

"tag_base" property in service definition must exists

This a programming error that we should not except to happen.

Contact Devo Support.

10

"tag_base" property in service definition must be a string

This a programming error that we should not except to happen.

Contact Devo Support.

11

"http_status_valid_codes" property in service definition must exists.