Page Comparison

Configuration requirements

To run this collector, there are some configurations detailed below that you need to consider.

...

Configuration

...

Details

...

Azure account

...

Azure account with admin level permissions and Azure AD tenant.

...

Credentials

...

The credentials configuration block has been filled correctly.

More information

Overview

The Microsoft Graph Collector provides the ability to collect data and intelligence from services such as Microsoft 365, Windows, and Enterprise Mobility and Security. This Currently, this data collector is able to ingest only security alerts, scores, provisioning, audit , and sign - ins retrieved from the Microsoft products, allowing you to empower streamlined . This empowers customers to streamline security operations and better defend against increasing cyber threats faced in their Azure AD and Microsoft 365 environments and beyond.

Devo collector features

...

Feature

...

Details

...

Allow parallel downloading (multipod)

...

• Allowed

...

Running environments

...

Collector server

...

Devo’s Microsoft Graph collector also enables customers to correlate events and context to improve threat protection and response, and includes key entities described in the next sections.

Configuration requirements

To run this collector, there are some configurations detailed below that you need to consider.

Devo collector features

Data sources

Vendor setup

...

Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:

1. An Azure account that has an active subscription.

2. The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).

3. A working Azure AD tenant.

You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.

API limits, delays and known issues

The Graph API imposes some throttling limits to the applications. The number of requests in a given time is limited according to several criteria. There is a generic limitation for the Graph API and service specific limits.

Any request can be evaluated against multiple limits, depending on the scope of the limit (per app across all tenants, per tenant for all apps, per app per tenant, and so on), the request type (GET, POST, PATCH, and so on), and other factors. The first limit to be reached triggers throttling behavior.

In the case of the services used by the collector, the limits are:

A complete reference of the limits can be found here.

When the collector reaches the API limit, the API returns a 429 error code. This is reflected in the logs of the collector with a message like 429 Error during API call. If several of these messages are found, the solution is to use the request_period_in_seconds for this service to decrease the frequency of calls.
For instance, a request period of 300 means that the API will be called every 5 minutes for this service, instead of the default value of 60 seconds.

API delays

Microsoft SLAs can be anywhere from 3 minutes to 6 hours in most cases. Check more information here.

Vendor setup

Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:

1. An Azure account that has an active subscription.

2. The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).

3. A working Azure AD tenant.

You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.

...

...

...

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Use the following command to add the Docker image to the system:

Rw ui tabs macro
Rw tab title On-premise collector This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. Structure The following directory structure should be created for being used when running the collector: Code Block <any_directory> └── devo-collectors/ └── <product_name>/ ├── certs/ │ ├── chain.crt │ ├── <your_domain>.key │ └── <your_domain>.crt ├── state/ └── config/ └── config.yaml Note Replace <product_name> with the proper value. Devo credentials In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here. Note Replace <product_name> with the proper value. Editing the config.yaml file Code Block globals: debug: false id: <short<collector_uniqueid_id>value> name: microsoft<collector_name_graphvalue> persistence: type: filesystem config: directory_name: state outputs: devo_us_1: type: devo_platform config: address: collector-us.devo.io # address: collector-eu.devo.io<devo_address> port: 443 type: SSL chain: chain.crt<chain_filename> cert: <devo<cert_domain.crt>filename> key: <devo<key_domain.key>filename> inputs: microsoft_graph_audit: id: 1<short_unique_id> enabled: true request_per_second: 5 credentials: tenant_id: <tenant_id_value> client_id: <client_id_value> client_secret: <client_secret_value> servicesenvironment: <environment_value> audit: override_top_level_domain: <override_top_level_domain_value> request_period_in_seconds: <request_period_in_secondsoverride_base_url: <override_base_url_value> reset_persistence_authoverride_login_url: <reset<override_persistencelogin_authurl_value> start_time: <start_time_value> override_scope_value: <override_scope_value_value> services: override_time_deltadirectory_audits: request_period_in_daysseconds: <override<request_time_deltaperiod_in_daysseconds_value> msstart_time_365in_environmentutc: <ms<start_time_365in_environmentutc_value> customadditional_endpointsfilter: <additional_filter_value> override_base_url_maintag: <override_base_url_maintag_value> override_query_basewindow_urlmax_vendorseconds: <override_basequery_window_urlmax_vendorseconds_value> provisioning_audits: override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value> request_period_in_seconds: <request_period_in_seconds_value> start_time_in_utc: <start_time_in_utc_value> additional_filter: <additional_filter_value> override_login_urltag: <override_login_urltag_value> override_scope_query_window_max_seconds: <override_scopequery_window_max_seconds_value> provisioningsignIns: request_period_in_seconds: <request_period_in_seconds_value> resetstart_time_persistencein_authutc: <reset<start_time_persistencein_authutc_value> startadditional_timefilter: <start<additional_timefilter_value> override_time_delta_in_daystag: <override_time_delta_in_days_tag_value> ms_365_environment: <ms_365_environmentoverride_query_window_max_seconds: <override_query_window_max_seconds_value> customsignIns_endpointsv2: overriderequest_baseperiod_urlin_mainseconds: <override<request_baseperiod_urlin_mainseconds_value> overridestart_basetime_urlin_vendorutc: <override<start_basetime_urlin_vendorutc_value> override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value> additional_filter: <additional_filter_value> override_login_urltag: <override_login_urltag_value> override_scopeoverride_query_window_max_seconds: <override_scopequery_window_max_seconds_value> microsoft_graph_security: signInid: <short_unique_id> enabled: true request_period_in_seconds: <request_period_in_seconds_value> credentials: reset_persistence_auth: <reset_persistence_authtenant_id: <tenant_id_value> startclient_timeid: <start<client_timeid_value> override_time_delta_in_daysclient_secret: <override<client_time_delta_in_days_value> secret_value> ms_365_environment: <ms_365<environment_environment_value> custom_endpoints: override_top_level_domain: <override_top_level_domain_value> override_base_url_main: <override_base_url_main_value> override_baselogin_url_vendor: <override_baselogin_url_vendor_value> override_base_url_vendor_with_sub_providerscope_value: <override_base_url_vendor_with_sub_providerscope_value_value> services: override_login_url: <override_login_url_value>secure_score_control_profiles: request_period_in_seconds: <request_period_in_seconds_value> additional_filter: <additional_filter_value> override_scopetag: <override_scopetag_value> signInsecure_nonInteractivescores: request_period_in_seconds: <request_period_in_seconds_value> resetstart_time_persistencein_authutc: <reset<start_time_persistencein_authutc_value> startadditional_timefilter: <start<additional_timefilter_value> override_time_delta_in_daystag: <override_time_delta_in_days_tag_value> ms_365_environment: <ms_365_environmentoverride_query_window_max_seconds: <override_query_window_max_seconds_value> custom_endpointsalerts: overriderequest_baseperiod_urlin_mainseconds: <override<request_baseperiod_urlin_mainseconds_value> overridestart_basetime_urlin_vendorutc: <override<start_basetime_urlin_vendorutc_value> override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provideradditional_filter: <additional_filter_value> override_login_urltag: <override_logintag_url_value> override_scope_query_window_max_seconds: <override_scopequery_window_max_seconds_value> signInalerts_servicePrincipalv2: request_period_in_seconds: <request_period_in_seconds_value> resetstart_time_persistencein_authutc: <reset<start_time_persistencein_authutc_value> startadditional_timefilter: <start<additional_timefilter_value> override_time_delta_in_daystag: <override_time_delta_in_days_tag_value> ms_365_environmentoverride_query_window_max_seconds: <ms_365_environment_value> custom_endpoints: override_base_url_main: <override_base_url_main_value> override_base_url_vendor: <override_base_url_vendor_value> override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value> override_login_url: <override_login_url_value> override_scope: <override_scope_value> signIn_managedIdentity: request_period_in_seconds: <request_period_in_seconds_value> reset_persistence_auth: <reset_persistence_auth_value> start_time: <start_time_value> override_time_delta_in_days: <override_time_delta_in_days_value> ms_365_environment: <ms_365_environment_value> custom_endpoints: override_base_url_main: <override_base_url_main_value> override_base_url_vendor: <override_base_url_vendor_value> override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value> override_login_url: <override_login_url_value> override_scope: <override_scope_value> secure_score_control_profile: tag_version: v2 request_period_in_seconds: <request_period_in_seconds_value> start_time: <start_time_value> ms_365_environment: <ms_365_environment_value> custom_endpoints: override_base_url_main: <override_base_url_main_value> override_base_url_vendor: <override_base_url_vendor_value> override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value> override_login_url: <override_login_url_value> override_scope: <override_scope_value> alerts: request_period_in_seconds: <request_period_in_seconds_value> reset_persistence_auth: <reset_persistence_auth_value> start_time: <start_time_value> override_time_delta_in_days: <override_time_delta_in_days_value> ms_365_environment: <ms_365_environment_value> custom_endpoints: override_base_url_main: <override_base_url_main_value> override_base_url_vendor: <override_base_url_vendor_value> override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value> override_login_url: <override_login_url_value> override_scope: <override_scope_value> secure_scores: tag_version: v2 request_period_in_seconds: <request_period_in_seconds_value> reset_persistence_auth: <reset_persistence_auth_value> start_time: <start_time_value> override_time_delta_in_days: <override_time_delta_in_days_value> ms_365_environment: <ms_365_environment_value> custom_endpoints: override_base_url_main: <override_base_url_main_value> override_base_url_vendor: <override_base_url_vendor_value> override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value> override_login_url: <override_login_url_value> override_scope: <override_scope_value> alerts_v2: request_period_in_seconds: <request_period_in_seconds_value> reset_persistence_auth: <reset_persistence_auth_value> start_time: <start_time_value> override_time_delta_in_days: <override_time_delta_in_days_value> ms_365_environment: <ms_365_environment_value> custom_endpoints: override_base_url_main: <override_base_url_main_value><override_query_window_max_seconds_value> Info All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. Replace the placeholders with your required values following the description table below: Parameter Data type Type Value Range / Format Details collector_id_value str mandatory minimum length: 1 maximum length: 5 Use this parameter to give a unique ID to this collector. collector_name_value str mandatory minimum length: 1 maximum length: 10 Use this parameter to give a name to this collector. devo_address str mandatory One of: collector-us.devo.io collector-eu.devo.io Use this parameter to identify the Devo Cloud where the events will be sent. chain_filename str mandatory minimum length: 4 maximum length: 20 Use this parameter to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt. cert_filename str mandatory minimum length: 4 maximum length: 20 Use this parameter to identify the file.cert downloaded from your Devo domain. key_filename str mandatory minimum length: 4 maximum length: 20 Use this parameter to identify the file.key downloaded from your Devo domain. short_unique_id str mandatory minimum length: 5 maximum length: - Use this parameter to give a unique ID to this input service. This parameter is used to build the persistence address; do not use the same value for multiple collectors. It could cause a collision. tenant_id_value str mandatory minimum length: 1 This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. client_id_value str mandatory minimum length: 1 This is the Application (client) ID you created in Azure AD. You can obtain it from the Overview page in your registered application. client_secret_value str mandatory minimum length: 1 This is the client secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. environment_value str optional minimum length: 1 This is an optional control parameter that is injected into the events and allows you to differentiate the environment. For example: dev and prod. Info This parameter should be removed if it is not used. request_period_in_seconds_value int optional minimum length: 60 Period in seconds used between each data pulling. This value will overwrite the default value (300 seconds). start_time_in_utc_value str optional UTC datetime string having datetime string format %Y-%m-%dT%H-%M-%SZ (e.g., “2020-01-01T00:00:01Z”) This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (one month back for example) before downloading new events. Info This parameter should be removed if it is not used. additional_filter_value str optional minimum length: 1 This parameter allows you to add an additional Microsoft Graph OData query clause to the default filters. By default, the collector attempts to fetch all data possible for a given Microsoft Graph entity type. See OData query parameters documentation for more information on how to format filters for Microsoft Graph API. Info This parameter should be removed if it is not used. override_tag_value str optional Devo tag-friendly string (no special characters, spaces, etc.) For more information see Devo Tags An optional tag that allows users to override the service default tags. Info This parameter should be removed if it is not used. override_top_level_domain_value str optional One of: com us Default value:com This is an optional parameter that allows you to override the default top level domain. If you're working in a Microsoft 365 GCC environment, continue using the worldwide endpoints (com). If you're working in a Microsoft 365 GCC High environment, use us. Info This parameter should be removed if it is not used. override_base_url_value str optional default value: https://graph.microsoft.{top_level_domain} A parameter that allows you to override the base URL. Info This parameter should be removed if it is not used. override_login_url_value str optional default value: https://login.microsoftonline.com/ A parameter that allows you to override the login URL. Info This parameter should be removed if it is not used. override_scope_value str optional default value: https://graph.microsoft.com/.default A parameter that allows you to override the scope value. Info This parameter should be removed if it is not used. override_query_window_max_seconds int optional default value: None A parameter that allows you to override the maximum query window. This is useful for any APIs that greatly restrict the amount of data one can fetch for a given datetime range (performance reasons). The collector will continuously fetch data but in smaller increments. Info This parameter should be removed if it is not used. Download the Docker image The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: Collector Docker image SHA-256 hash collector-ms-graph-collector-if-docker-image-2.0.1 0dcd32b48f769fd597f61edb76f83086f8fcfc16dc8003b9e6a68388a031ddd3 Use the following command to add the Docker image to the system: Code Block gunzip -c <collector-ms-graph-collector-if-docker-image>-<1.2.0>.tgz | docker load Note Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value. The Docker image can be deployed on the following services: Docker Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/ Code Block docker run --name collector-<product_name> --volume $PWD/certs:/devo-collector/certs --volume $PWD/config:/devo-collector/config --volume $PWD/state:/devo-collector/state --env CONFIG_FILE=config.yaml --rm --interactive --tty <image_name>:<version> Note Replace <product_name>, <image_name> and <version> with the proper values. Docker Compose The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory. Code Block version: '3' services: collector-<product_name>: image: <image_name>:${IMAGE_VERSION:-latest} container_name: collector-<product_name> volumes: - ./certs:/devo-collector/certs - ./config:/devo-collector/config - ./credentials:/devo-collector/credentials - ./state:/devo-collector/state environment: - CONFIG_FILE=${CONFIG_FILE:-config.yaml} To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory: Code Block IMAGE_VERSION=<version> docker-compose up -d Note Replace <product_name>, <image_name> and <version> with the proper values. Rw tab title Cloud collector We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer: In the Collector Server GUI, access the domain in which you want this instance to be created Click Add Collector and find the one you wish to add. In the Version field, select the latest value. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain). In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups. In the Parameters section, establish the Collector Parameters as follows below: Editing the JSON configuration Code Block { "global_overrides": { "debug": false }, "inputs": { "microsoft_graph_audit": { "id": "<short_unique_id>", "enabled": true, "credentials": { "tenant_id": "<tenant_id_value>", "client_id": "<client_id_value>", "client_secret": "<client_secret_value>" }, "environment": "<environment_value>", "override_top_level_domain": "<override_top_level_domain_value>", "override_base_url": "<override_base_url_value>", "override_login_url": "<override_login_url_value>", "override_scope_value": "<override_scope_value_value>", "services": { "directory_audits": { override"request_baseperiod_urlin_vendorseconds": <override"<request_baseperiod_urlin_vendorseconds_value>", override"start_basetime_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>in_utc": "<start_time_in_utc_value>", "additional_filter": "<additional_filter_value>", "override_login_urltag": "<override_logintag_url_value>", "override_scopequery_window_max_seconds": "<override_scope_value> Info All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. Replace the placeholders with your required values following the description table below:
Parameter	Data type	Type	Value range / Format	Details
<debug_status>	bool	Mandatory	false / true	If the value is true, the debug logging traces will be enabled when running the collector. If the value is false, only the info, warning and error logging levels will be printed.
<collector_id>	int	Mandatory	Minimum length: 1 Maximum length: 5	Use this param to give a unique id to this collector.
<collector_name>	str	Mandatory	Minimum length: 1 Maximum length: 10	Use this param to give a valid name to this collector, like microsoft_graph.
<devo_address>	str	Mandatory	collector-us.devo.io collector-eu.devo.io	Use this param to identify the Devo Cloud where the events will be sent.
<chain_filename>	str	Mandatory	Minimum length: 4 Maximum length: 20	Use this param to identify the chain.cert  file downloaded from your Devo domain. Usually this file's name is: chain.crt
<cert_filename>	str	Mandatory	Minimum length: 4 Maximum length: 20	Use this param to identify the file.cert downloaded from your Devo domain.
<key_filename>	str	Mandatory	Minimum length: 4 Maximum length: 20	Use this param to identify the file.key downloaded from your Devo domain.
<input_id>	int	Mandatory	Minimum length: 1 Maximum length: 5	Use this param to give a unique id to this input service. Note This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
<run_service>	bool	Mandatory	false / true	Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.
<requests_per_second_value>	int	Optional	Minimum value: 1	Customize the maximum number of API requests per second. If not used, the default setting will be used: 60 requests/sec. Info This parameter should be removed itf it is not used.
<tenant_id_value>	str	Mandatory	UUID format	This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.
<client_id_value>	srt	Mandatory	UUID format	This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.
<client_secret_value>	srt	Mandatory	Any non-whitespace character	This is the Client’s secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application.
<request_period_in_seconds_value>	int	Optional	Minimum value: 1	The amount (in seconds) in which the service’s collection is scheduled. Info This parameter should be removed itf it is not used.
<start_time_value>	srt	Optional	For the secure_scores service: YYYY-MM-DDTHH:mm:ssZ For the rest of the services: YYYY-MM-DDTHH:mm:ss.SSSZ	This will allow you to start from a specific date in case you want to ingest historic events. If not set, it will start at the current time. Info This parameter should be removed itf it is not used.
<tag_version>	srt	Optional	Only accepts v1 (default) or v2	The tag_version parameter is only implemented for the alerts, secure_scores and secure_score_control_profile(s) services. This parameter configured the destination tables according to their tagging: v1 will use the original cloud.msgraph... tagging. v2 will use the new tags cloud.azure... and cloud.office365... introduced in the v1.1.3 release. The detailed table destination depending on the tag_version's value is available in the Devo categorization and destination section. Info It is not necessary to include this parameter in the configuration to maintain backward compatibility. If there is no value, "v1" is assumed to maintain backward compatibility. With this, we can upgrade collectors whose version is previous to v1.1.3 without applying any changes. However, the recommendation is to set it to "v2" for new deployments. Info This parameter should be removed if it is not used.
Collector Docker image	SHA-256 hash
collector-ms-graph-collector-if-docker-image-2.0.0	ceb50ad407755a92d4476be3070e574abb8fe1e6b6a91bb96c855e865ff9bfb8

gunzip -c <collector-ms-graph-collector-if-docker-image>-<1.2.0>.tgz | docker load

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

IMAGE_VERSION=<version> docker-compose up -d

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Alerts

Alerts are potential security issues within a customer's tenant that Microsoft or partner security solutions have identified and flagged for action or notification. With the Microsoft Graph alerts entity, you can unify and streamline the management of security issues across all integrated solutions.

Alerts Security Providers:

• Microsoft Defender for Cloud

• Azure Active Directory Identity Protection

• Microsoft Defender for Cloud Apps

• Microsoft Defender for Endpoint

• Microsoft Defender for Identity

• Microsoft 365

• Azure Information Protection

• Azure Sentinel

Secure Scores

Microsoft Secure Score is a security analytics solution that gives you visibility into your security portfolio and how to improve it. With a single score, you can better understand what you have done to reduce your risk in Microsoft solutions. You can also compare your score with other organizations and see how your score has been trending over time. The Microsoft Graph secureScore and secureScoreControlProfile entities help you balance your organization's security and productivity needs while enabling the appropriate mix of security features. You can also project what your score would be after you adopt security features.

Azure Active Directory reports (Sign-in, Audit, Provisioning)

Azure Active Directory (Azure AD) reports providing a comprehensive view of activity in your environment. The provided data enables you to:

• Determine how your apps and services are utilized by your users.

• Detect potential risks affecting the health of your environment.

• Troubleshoot issues preventing your users from getting their work done.

These reports help you understand the behavior of users in your organization. There are three types of reports that this collector pulls from Azure AD:

• Sign-ins: Information about sign-ins and how your resources are used by your users.

• Audit: Information about changes applied to your tenants such as users and group management or updates applied to your tenant’s resources.

• Provisioning: Activities performed by the provisioning service, such as the creation of a group in ServiceNow or a user imported from Workday.

Devo categorization and destination

Here you can see how each service that has configurable tagging will tag its events depending on the value of the tag_version parameter:

...

Service

...

Old tagging (v1)

...

New tagging (v2)

...

secure_scores

...

cloud.msgraph.security.secure_scores.1

...

cloud.office365.security.scores.1.msgraph

...

secure_score_control_profiles

...

cloud.msgraph.security.secure_score_control_profiles.1

...

cloud.office365.security.scorecontrol.1.msgraph

...

alerts

...

cloud.msgraph.security.alerts.1

...

The tagging is based on the provider. See the next table.

The alerts Service Now uses dynamic tagging based on the event’s provider field. This is the provider/tag correspondence:

...

Provider

...

New tagging (v2)

...

IPC

...

cloud.azure.ad.alerts.1.msgraph

...

MCAS

...

cloud.office365.cloud_apps.alerts.1.msgraph

...

Microsoft Defender ATP

...

cloud.office365.security.alerts.1.msgraph

...

Office 365 Security and Compliance

...

cloud.azure.sentinel.alerts.1.msgraph

...

Azure Sentinel

...

cloud.office365.identity.alerts.1.msgraph

...

Azure Advanced Threat Protection

...

cloud.azure.securitycenter.alerts.1.msgraph

Events service

...

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

...

Component

...

Description

...

Setup

...

The setup module is in charge of authenticating the service and managing the token expiration when needed.

...

Puller

...

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Access Token has been validated successfully
INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Setup for module <MicrosoftGraphNonTimeBasedPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 2 requests, retrieved all vendor list, detected 1 unique vendors
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> MicrosoftGraphNonTimeBasedVendorPuller(microsoft_graph#1,secure_score_control_profile#predefined,MicrosoftGraphNonTimeBasedPuller#SecureScore.None) -> Starting thread
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Data collection completed. Elapsed time: 4.708 seconds. Waiting for 55.292 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms

...

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Common for all services

...

Error type

...

Error ID

...

Error message

...

Cause

...

Solution

...

MicrosoftGraphPullerCredentialsException

...

1

...

Invalid tenant. This may happen if there are no active subscriptions for the tenant

...

The tenant is not valid according to Microsoft AD. Probably, the setup has not been correctly followed.

...

Revisit the setup instructions and check that all the steps have been correctly followed.

...

1

...

Access Token validation has been failed, code: {status_code}, text: {text_str}

...

There was a problem obtaining the authentication token.

...

Read the exact error to understand what is the real cause.

...

2

...

Access Token not valid or client_id does not exist

...

There was an unknown problem during the authentication..

...

Check that the credentials have been correctly set up.

...

MicrosoftGraphPullerCreationException
ModuleDefinitionError

...

1

...

"module_properties" property in service definition must exists / "module_properties" mandatory property is missing or empty

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

2

...

"module_properties" property in service definition must be a dictionary

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

3

...

"resource_type" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

4

...

"resource_type" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

5

...

"base_url_main" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

6

...

"base_url_main" property in service definition must be a string.

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

7

...

"base_url_vendor" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

8

...

"base_url_vendor" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

9

...

"tag_base" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

10

...

"tag_base" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

11

...

"http_status_valid_codes" property in service definition must exists.

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

12

...

"http_status_valid_codes" property in service definition must be a list

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

13

...

"login_url" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

14

...

"login_url" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

15

...

"scope" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

16

...

"scope" property in service definition must be a string.

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

MicrosoftGraphPullerCreationException

...

37

...

"tag_version" property for {service_name} from configuration file is not valid, expected: v1, v2 or None

...

This configuration parameter expects a string with value v1, v2 or null but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string with value v1, v2 or null.

...

MicrosoftGraphPullerRetrieveException

...

1

...

Not defined "next_page_url"

...

When retrieving data from the MS Graph API, we expect that each paginated response has a next_page_url property in its JSON. This error occurs when this property was not found.

...

Enable debug mode and inspect the requested URLs. Try to replicate those to see the response obtained.

...

2

...

Response text: "{response_text}", HTTP error: {http_error}

...

When retrieving data from the MS Graph API, we expect that the response code is within the [200-400) range; otherwise (HTTP response code ≥ 400), it will raise this error.

...

The error will include the response’s text. This should tell you what the problem is. The solution will depend on the type of the problem.

...

MicrosoftGraphPullerConnectionException

...

3

...

Operation timed out: {error_message}

...

When retrieving data from the MS Graph API, the server took too long to respond and the connection was closed.

...

Check your network connection and that the MS Graph API is operative.

...

4

...

Connection error detected: {error_message}

...

Some other error regarding the connection with the MS Graph API server occurred.

...

You should read the actual error message to understand the underlying issue and know how to solve it.

...

MicrosoftGraphPullerRetrieveException

...

5

...

{error_message}

...

Some unknown error occurred.

...

You should read the actual error message to understand the underlying issue and know how to solve it.

Non-time-base services (secure_score_control_profiles)

...

Error type

...

Error ID

...

Error message

...

Cause

...

Solution

...

MicrosoftGraphPullerCreationException

...

17

...

"request_max_items_per_request" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

18

...

"request_max_items_per_request" property in service definition must be an intege

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

19

...

"requests_per_minute" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

20

...

"requests_per_minute" property in service definition must be an integer

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

21

...

"requests_per_minute" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

22

...

"requests_per_minute" property in service definition must be an integer

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

23

...

"new_service_name" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

27

...

"tag" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

28

...

"credentials" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

29

...

"credentials" property in configuration must be a dictionary

...

This configuration parameter expects a JSON object value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a JSON object.

...

30

...

"tenant_id" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

31

...

"tenant_id" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

32

...

"client_id" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

33

...

"client_id" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

34

...

"client_secret" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

35

...

"client_secret" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

Time-based (alerts, secure_scores) & audit (audit, provisioning, signIn, signIn_nonInteractive, signIn_servicePrincipal, signIn_managedIdentity) services

...

Error Type

...

Error ID

...

Error Message

...

Cause

...

Solution

...

ModuleDefinitionError

...

5

...

"base_url_main_only_first_page" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

6

...

"base_url_main_only_first_page" property in service definition must be a boolean

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

7

...

"base_url_vendor_with_sub_provider" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

8

...

"base_url_vendor_with_sub_provider" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

17

...

"base_url_main_items_per_request" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

18

...

"base_url_main_items_per_request" property in service definition must be an integer

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

19

...

"base_url_main_items_per_request" property in service definition must be a positive value

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

17

...

"base_url_vendor_items_per_request" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

18

...

"base_url_vendor_items_per_request" property in service definition must be an integer

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

19

...

"base_url_vendor_items_per_request" property in service definition must be a positive value

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

20

...

"max_result_set_size" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

21

...

"max_result_set_size" property in service definition must be an integer

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

22

...

"max_result_set_size" property in service definition must be a positive value

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

24

...

"legacy_provider_mapping_old_new" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

25

...

"requests_per_minute" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

26

...

"requests_per_minute" property in service definition must be an integer

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

27

...

"requests_per_minute" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

28

...

"requests_per_minute" property in service definition must be an integer

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

29

...

"timestamp_field" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

30

...

"timestamp_field" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

31

...

"start_time_regex" property in service definition must exists

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

32

...

"start_time_regex" property in service definition must be a string

...

This a programming error that we should not except to happen.

...

Contact Devo Support.

...

InputConfigurationError

...

1

...

"microsoft_graph" mandatory property is missing or empty

...

The input configuration is missing.

...

Ensure that the configuration includes an input configuration.

...

2

...

"microsoft_graph" property must be a dictionary

...

The input configuration expects to have a JSON object value, but it has parsed a different type of value.

...

Ensure that the configuration for this input is a JSON object.

...

3

...

"credentials" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

4

...

"credentials" property in configuration must be a dictionary

...

This configuration parameter expects a JSON object value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a JSON object.

...

5

...

"tenant_id" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

6

...

"tenant_id" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

7

...

"client_id" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

8

...

"client_id" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

9

...

"client_secret" property in configuration must exists

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

10

...

"client_secret" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

ServiceConfigurationError

...

1

...

"{service_name}" mandatory property is missing or empty

...

This configuration parameter is missing.

...

Ensure that this parameter is present and has a value.

...

ServiceConfigurationError

...

2

...

"{service_name}" property must be a dictionary

...

This configuration parameter expects a JSON object value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a JSON object.

...

MicrosoftGraphPullerCreationException

...

27

...

"tag" property in configuration must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

36

...

"start_time" property in service definition must be a string

...

This configuration parameter expects a string value, but it has parsed a different type of value.

...

Ensure that the value for this parameter is a string.

...

37

...

"start_time" property from configuration file format is not valid, expected: "{start_time_regex}"

...

This configuration parameter expects a date that matches the indicated regular expression, but it did not match.

...

Ensure that the value for this parameter is a valid date according to the indicated regular expression.

...

38

...

"tag_version" property for {self.service_name} from configuration file is not valid, received "v2", but there is no "tag_base_v2" property in module definitions

...

You set the tag_version parameter to v2, but the service does not have a v2 tagging.

...

Contact Devo Support.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

...

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

   INFO MainProcess::MainThread -> Added "[collector_dir]" directory to the Python path
   INFO MainProcess::MainThread -> Added "[collector_dir]/config_internal" directory to the Python path
   INFO MainProcess::MainThread -> {"production_mode": false, "python_version": "3.9.12 (main, Apr  6 2022, 10:19:29) \n[Clang 12.0.5 (clang-1205.0.22.9)]", "current_directory": "[collector_dir]", "exists_config_dir": true, "exists_config_internal_dir": true, "exists_certs_dir": true, "exists_credentials_dir": true}
   INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-test-local.yaml", "job_config_loc": null, "collector_config_loc": null}
   INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
   INFO MainProcess::MainThread -> "/etc/devo/job" does not exists
   INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
   INFO MainProcess::MainThread -> "/etc/devo/collector" does not exists
   INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "[collector_dir]/config/config-test-local.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
   INFO MainProcess::MainThread -> {"build_time": "UNKNOWN", "os_info": "macOS-12.5.1-x86_64-i386-64bit", "collector_name": "microsoft_graph", "collector_version": "1.2.0", "collector_owner": "integrations_factory@devo.com", "started_at": "2022-09-01T14:57:53.453415Z"}
   INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 10000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
   INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-us.devo.io', 'port': 443, 'type': 'SSL', 'chain': 'chain.cer', 'cert': 'cert.cer', 'key': 'key.key', 'concurrent_connections': 1, 'period_sender_stats_in_seconds': 300, 'activate_final_queue': False, 'threshold_for_using_gzip_in_transport_layer': 1.1, 'compression_level': 6, 'compression_buffer_in_bytes': 51200, 'generate_metrics': False}}}
   INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
   INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
   INFO InputProcess::MainThread -> Process Started
   INFO OutputProcess::MainThread -> Process started
   INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
   INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
   INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
   INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
   INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
   INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
   INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
   INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
   INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
   INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
   INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
   INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
   INFO OutputProcess::MainThread -> [GC] global: 67.8% -> 67.8%, process: RSS(38.10MiB -> 38.16MiB), VMS(32.82GiB -> 32.82GiB)
   INFO InputProcess::MainThread -> InputThread(microsoft_graph,1) - Starting thread (execution_period=600s)
   INFO InputProcess::MainThread -> ServiceThread(microsoft_graph,1,secure_score_control_profile,predefined) - Starting thread (execution_period=600s)
   INFO InputProcess::MainThread -> MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Starting thread
   INFO InputProcess::MainThread -> MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) - Starting thread
WARNING InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Waiting until setup will be executed
   INFO InputProcess::MainThread -> [GC] global: 67.8% -> 67.8%, process: RSS(38.35MiB -> 38.35MiB), VMS(32.70GiB -> 32.70GiB)
   INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-us.devo.io:443", "chain_path": "[collector_dir]/certs/chain.cer", "cert_path": "[collector_dir]/certs/cert.cer", "key_path": "[collector_dir]/certs/key.key", "transport_layer_type": "SSL"}, hostname: "2019EMEA0386.local", session_id: "4532931840"

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

...

Sender services

...

Description

...

internal_senders

...

In charge of delivering internal metrics to Devo such as logging traces or metrics.

...

standard_senders

...

In charge of delivering pulled events to Devo.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

...

Logging trace

...

Description

...

Number of available senders: 1

...

Displays the number of concurrent senders available for the given Sender Service.

...

sender manager internal queue size: 0

...

Displays the items available in the internal sender queue.

...

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)

...

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

• 44 events were sent to Devo since the collector started.

• The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.

• 21 events where sent to Devo between the last UTC checkpoint and now.

• Those 21 events required 0.007 seconds to be delivered.

INFO InputProcess::MainThread -> [GC] global: 67.8% -> 67.8%, process: RSS(38.35MiB -> 38.35MiB), VMS(32.70GiB -> 32.70GiB)
INFO OutputProcess::MainThread -> [GC] global: 67.8% -> 67.8%, process: RSS(38.10MiB -> 38.16MiB), VMS(32.82GiB -> 32.82GiB)

query_window_max_seconds_value>"
        },
        "provisioning_audits": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "signIns": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "signIns_v2": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        }
      }
    },
    "microsoft_graph_security": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "tenant_id": "<tenant_id_value>",
        "client_id": "<client_id_value>",
        "client_secret": "<client_secret_value>"
      },
      "environment": "<environment_value>",
      "override_top_level_domain": "<override_top_level_domain_value>",
      "override_base_url": "<override_base_url_value>",
      "override_login_url": "<override_login_url_value>",
      "override_scope_value": "<override_scope_value_value>",
      "services": {
        "secure_score_control_profiles": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>"
        },
        "secure_scores": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "alerts": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        },
        "alerts_v2": {
          "request_period_in_seconds": "<request_period_in_seconds_value>",
          "start_time_in_utc": "<start_time_in_utc_value>",
          "additional_filter": "<additional_filter_value>",
          "override_tag": "<override_tag_value>",
          "override_query_window_max_seconds": "<override_query_window_max_seconds_value>"
        }
      }
    }
  }
}

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Restart the persistence for a service

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

1. Edit the configuration file.

2. Change the value of the start_time_in_utc parameter to a different one.

3. Save the changes.

4. Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Troubleshooting

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Common logic

Collector operations

Verify collector operations

2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-test-local.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> "\etc\devo\job" does not exists
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> "\etc\devo\collector" does not exists
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "C:\git\collectors2\devo-collector-<name>\config\config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
2023-01-10T15:22:57.171 WARNING MainProcess::MainThread -> [WARNING] Illegal global setting has been ignored -> multiprocessing: False

2023-01-10T15:23:00.788 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting
thread
2023-01-10T15:23:00.789 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) ->
Starting thread (every 300 seconds)
2023-01-10T15:23:00.790 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) ->
Starting thread
2023-01-10T15:23:00.842 INFO OutputProcess::MainThread -> global_status: {"output_process": {"process_id": 18804,
"process_status": "running", "thread_counter": 21, "thread_names": ["MainThread", "pydevd.Writer", "pydevd.Reader",
"pydevd.CommandThread", "pydevd.CheckAliveThread", "DevoSender(standard_senders,devo_sender_0)",
"DevoSenderManagerMonitor(standard_senders,devo_1)", "DevoSenderManager(standard_senders,manager,devo_1)",
"OutputStandardConsumer(standard_senders_consumer_0)",

Check memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

• The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

• The global pressure of the available memory is displayed in the global value.

• All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB ->
 410.02MiB)
 INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB ->
 705.28MiB)Change log

1.X to 2.X migration guide

Version 2.0.0 introduced several changes to the collector's configuration. When upgrading the collector, users must make changes to their configuration file to ensure that the collector continues to work as expected.

In versions 1.X of the collector, some services had a config parameter tag_version with values v1or v2. The effect of this parameter is that the destination table for these services will be different. These are the destination tables according to the tag_version value:

In the new collector 2.0.0, the old config parameter tag_version has been removed. The same effect as v1can be made using override_tag, with these values:

For instance, if we have this in the old config:

secure_scores:
        tag_version: v1
      secure_score_control_profile:
        tag_version: v1

you should change it to:

secure_scores:
        override_tag: cloud.msgraph.security.scores.2
      secure_score_control_profile:
        override_tag: cloud.msgraph.security.scorecontrol.2

In old versions of this collector, all alerts were sent to the table cloud.msgraph.security.alerts

However, as there are alerts of several types, in version 2.0.0 alerts are categorized according to their vendorInformation-provider field (their vendor) and sent to different tables:

It is possible to avoid this feature and send all alerts to the same table by editing the config file and changing the tag:

alerts:
        override_tag: cloud.msgraph.security.alerts.2

Config file comparison

Below you will find a sample 1.7.1 configuration file and a sample 2.0.0 configuration file.

inputs:
  microsoft_graph:
    id: <short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    services:
      audit:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      provisioning:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn_nonInteractive:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn_servicePrincipal:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      signIn_managedIdentity:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      secure_score_control_profile:
        tag_version: v2
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      alerts:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        pull_sliding_window_timespan_period: <pull_sliding_window_timespan_period_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      secure_scores:
        tag_version: v2
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        pull_sliding_window_timespan_period: <pull_sliding_window_timespan_period_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>
      alerts_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        start_time: <start_time_value>
        override_time_delta_in_days: <override_time_delta_in_days_value>
        ms_365_environment: <ms_365_environment_value>
        custom_endpoints:
          override_base_url_main: <override_base_url_main_value>
          override_base_url_vendor: <override_base_url_vendor_value>
          override_base_url_vendor_with_sub_provider: <override_base_url_vendor_with_sub_provider_value>
          override_login_url: <override_login_url_value>
          override_scope: <override_scope_value>

inputs:
  microsoft_graph_audit:
    id: <short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    environment: <environment_value>
    override_top_level_domain: <override_top_level_domain_value>
    override_base_url: <override_base_url_value>
    override_login_url: <override_login_url_value>
    override_scope_value: <override_scope_value_value>
    services:
      directory_audits:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      provisioning_audits:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      signIns:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      signIns_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
  microsoft_graph_security:
    id: <short_unique_id>
    enabled: true
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    environment: <environment_value>
    override_top_level_domain: <override_top_level_domain_value>
    override_base_url: <override_base_url_value>
    override_login_url: <override_login_url_value>
    override_scope_value: <override_scope_value_value>
    services:
      secure_score_control_profiles:
        request_period_in_seconds: <request_period_in_seconds_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
      secure_scores:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      alerts:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>
      alerts_v2:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        additional_filter: <additional_filter_value>
        override_tag: <override_tag_value>
        override_query_window_max_seconds: <override_query_window_max_seconds_value>

Config file changes

• The URL endpoints (override_base_url_main, override_base_url_vendor, override_base_url_vendor_with_sub_provider , override_login_url) have been moved from the individual services to the global configuration section.

• override_base_url_main has been renamed to override_base_url.

• tag_version has been removed.

• pull_sliding_window_timespan_period has been removed.

• reset_persistence_auth has been removed.

• override_time_delta_in_days has been removed.

• ms_365_environment has been replaced by override_top_level_domain. GCC High Gov should use us in the override_top_level_domain. Alternatively, users can use override_base_url to specify the GCC High Gov base URL.

• additional_filter has been added to all services. Users can use this field to specify additional filters that will be applied when querying the Microsoft Graph API.

• The collector can use new services from Graph (beta endpoint in Graph), that services, that use to be in a separate service for each type, have been consolidated into one service called signIns_v2. Users should remove all three services from their config and use only the signIns_v2 service.

• start_time has been renamed to start_time_in_utc.

Persistence changes

• The persistence object consists of the following fields: persistence_version, last_event_time_in_utc, last_ids, and next_link.

• The collector will automatically map old key names (e.g. last_polled_timestamp → last_event_time_in_utc) to the appropriate value.

Change log