Table of Contents | ||||
---|---|---|---|---|
|
Configuration requirements
To run this collector, there are some configurations detailed below that you need to consider.
...
Configuration
...
Details
...
Azure account
...
Azure account with admin level permissions and Azure AD tenant.
...
Credentials
...
The credentials configuration block has been filled correctly.
Info |
---|
Note |
If you are migrating from any 1.x version, please read this section. |
Overview
The Microsoft Graph Collector provides the ability to collect data and intelligence from services such as Microsoft 365, Windows, and Enterprise Mobility and Security. This Currently, this data collector is able to ingest only security alerts, scores, provisioning, audit , and sign - ins retrieved from the Microsoft products, allowing you to empower streamlined . This empowers customers to streamline security operations and better defend against increasing cyber threats faced in their Azure AD and Microsoft 365 environments and beyond.
Devo collector features
...
Feature
...
Details
...
Allow parallel downloading (multipod
)
...
Allowed
...
Running environments
...
Collector server
...
Devo’s Microsoft Graph collector also enables customers to correlate events and context to improve threat protection and response, and includes key entities described in the next sections.
Configuration requirements
To run this collector, there are some configurations detailed below that you need to consider.
Configuration | Details |
---|---|
Azure account | Azure account with admin level permissions and Azure AD tenant. |
Credentials | The credentials configuration block has been filled correctly. |
Info |
---|
More information Refer to the Vendor setup section to know more about these configurations. |
Devo collector features
Feature | Details |
---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events | Table
|
Flattening preprocessing No |
|
Allowed source events obfuscation |
|
Data sources
Data source | Description | API endpoint | Collector service name | Devo table |
---|
Available from release
Alerts
Represents potential security issues within a customer’s tenant that Microsoft or partner security solutions have identified.
Refer to Microsoft documentation about Alert Resource Type for more information.
Audit logs - provisioning | Represents an action performed by the Microsoft Entra provisioning service and its associated properties. |
|
alerts
Starting from v1.2.0, the destination table depends on the tag_version configuration parameter:
v1
: ºv2
: the destination will be dynamic, depending on the provider:
|
|
|
|
MCAS: cloud.office365.cloud_apps.alerts
.
Microsoft Defender ATP: cloud.office365.endpoint.alerts
.
Office 365 Security and Compliance: cloud.office365.security.alerts
.
Azure Sentinel: cloud.azure.sentinel.alerts
.
ASC: cloud.office365.identity.alerts
.
Azure Advanced Threat Protection: cloud.azure.securitycenter.alerts
.
For any other provider, it will fall back in v1
's tagging.
Info |
---|
The detailed table destination will depend on the tag_version’s value |
v1.0.0
Secure scores
Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held.
Refer to the Microsoft documentation for more information about Secure scores resources types.
https://graph.microsoft.com/v1.0/security/secureScores?$count=true&$filter=createdDateTime+ge+{start_time}+AND+vendorInformation/provider+eq+'{provider}'&$orderby=createdDateTime+asc,vendorInformation/provider+asc&$top={items_per_vendor_request}
secure_scores
Starting from v1.2.0, the destination table depends on the tag_version
configuration parameter:
v1
:cloud.msgraph.security.secure_scores
v2
:cloud.office365.security.scores
Info |
---|
The detailed table destination depending on the tag_version's value is available in the Devo categorization and destination section. |
v1.0.0
Secure score control profiles
Represents a tenant's secure score per control data.
Refer to the Microsoft documentation for more information about Secure score control profiles.
https://graph.microsoft.com/v1.0/security/secureScoreControlProfiles?$count=true
secure_score_control_profiles
Starting from v1.2.0, the destination table depends on the tag_version
configuration parameter:
v1
:cloud.msgraph.security.secure_score_control_profiles
v2
:cloud.office365.security.scorecontrol
Info |
---|
The detailed table destination depending on the tag_version's value is available in the Devo categorization and destination section. |
v1.0.0
Directory audit
Represents the directory audit items and its collection.
Refer to the Microsoft documentation for more information about Directory audit.
https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?$filter=activityDateTime ge {start_time}&$orderby=activityDateTime+asc&$top={items_per_main_request}
audit
cloud.azure.ad.audit
v1.2.0
Provisioning
Represents an action performed by the Azure AD Provisioning service and its associated properties.
Refer to the Microsoft documentation for more information about Provisioning.
https://graph.microsoft.com/beta/auditLogs/provisioning?$filter=activityDateTime ge {start_time}&$orderby=activityDateTime+asc&$top={items_per_main_request}
provisioning
cloud.azure.ad.audit
v1.2.0
Sign-in
Details user and application sign-in activity for a tenant (directory).
Refer to the Microsoft documentation for more information about Sign-in.
Note |
---|
These services return a huge volume of data. If the oldest available data is not especially relevant, it is recommended to set a close start time for the collector, to get the up-to-date state as soon as possible. |
signIn
:https://graph.microsoft.com/v1.0/auditLogs/signIns?$orderby=createdDateTime+asc&$top={items_per_main_request}
signIn_nonInteractive
:https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'nonInteractiveUser')&$orderby=createdDateTime+asc&$top={items_per_main_request}
signIn_servicePrincipal
:https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'servicePrincipal')&$orderby=createdDateTime+asc&$top={items_per_main_request}
signIn_managedIdentity
:https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'managedIdentity')&$orderby=createdDateTime+asc&$top={items_per_main_request}
signIn
signIn_nonInteractive
signIn_servicePrincipal
signIn_managedIdentity
signIn
:cloud.azure.ad.signin
signIn_nonInteractive
:cloud.azure.ad.noninteractive_user_signin
signIn_servicePrincipal
:cloud.azure.ad.service_principal_signin
signIn_managedIdentity
:cloud.azure.ad.managed_identity_signin
v1.2.0
Vendor setup
...
Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:
An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.
You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.
Note |
---|
You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits. |
Action
Steps
1
Register and configure the application
Go to Azure portal and click on Azure Active Directory.
Click on App registration on the left-menu side. Then click on + New registration.
On the Register and Application page:
Name the application.
Select Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) in Supported Accounts type.
In Redirect URI (optional) leave it as default (blank).
Click Register.
App registration page will open. Click on your app to configure it and give it permissions. You will see your app’s dashboard with information (docs, endpoints, etc.) when clicking it.
| ||||
Audit logs - directory | Represents the directory audit items and its collection. |
|
|
|
Audit logs - sign-ins | Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API. |
|
|
|
Audit logs - sign-ins (v2) | Details user and application sign-in activity for a tenant (directory). You must have a Microsoft Entra ID P1 or P2 license to download sign-in logs using the Microsoft Graph API. |
|
|
|
Legacy Alerts | This resource corresponds to the first generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft or a partner security solution has identified. This type of alerts federates calling of supported Azure and Microsoft 365 Defender security providers listed in Use the Microsoft Graph security API. It aggregates common alert data among the different domains to allow applications to unify and streamline management of security issues across all integrated solutions. |
|
|
|
Alerts (v2) | This resource corresponds to the latest generation of alerts in the Microsoft Graph security API, representing potential security issues within a customer's tenant that Microsoft 365 Defender, or a security provider integrated with Microsoft 365 Defender, has identified. When detecting a threat, a security provider creates an alert in the system. Microsoft 365 Defender pulls this alert data from the security provider, and consumes the alert data to return valuable clues in an alert resource about any related attack, impacted assets, and associated evidence. It automatically correlates other alerts with the same attack techniques or the same attacker into an incident to provide a broader context of an attack. Aggregating alerts in this manner makes it easy for analysts to collectively investigate and respond to threats. |
|
|
|
Secure Scores | Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held. This data is sorted by createdDateTime, from latest to earliest. This will allow you to page responses by using $top=n, where n = the number of days of data that you want to retrieve. |
|
|
|
Secure Scores Control Profiles | Represents a tenant's secure score per control data. By default, this resource returns all controls for a tenant and can explicitly pull individual controls. |
|
|
|
API limits, delays and known issues
The Graph API imposes some throttling limits to the applications. The number of requests in a given time is limited according to several criteria. There is a generic limitation for the Graph API and service specific limits.
Any request can be evaluated against multiple limits, depending on the scope of the limit (per app across all tenants, per tenant for all apps, per app per tenant, and so on), the request type (GET, POST, PATCH, and so on), and other factors. The first limit to be reached triggers throttling behavior.
In the case of the services used by the collector, the limits are:
Request type | Per app across all tenants |
---|---|
Any | 130,000 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 5 requests per 10 seconds |
| 150 requests per minute |
| 150 requests per minute |
| 150 requests per minute or 10,000 in a 10-minute period |
| 10,000 in a 10-minute period |
A complete reference of the limits can be found here.
When the collector reaches the API limit, the API returns a 429 error code. This is reflected in the logs of the collector with a message like 429 Error during API call
. If several of these messages are found, the solution is to use the request_period_in_seconds
for this service to decrease the frequency of calls.
For instance, a request period of 300 means that the API will be called every 5 minutes for this service, instead of the default value of 60 seconds.
API delays
Microsoft SLAs can be anywhere from 3 minutes to 6 hours in most cases. Check more information here.
Vendor setup
Anchor | ||||
---|---|---|---|---|
|
Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:
An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.
You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.
Note |
---|
You need the Admin level permissions on the Azure portal as the subscription setup will require admin consent API permissions, authentications, and audits. |
Action | Steps | |||
1 | Register and configure the application |
| ||
2 | Grant the required permissions |
| ||
3 | Obtain the requires credentials for the collector |
|
...
Collector service | Resource | Required permissions | Microsoft documentation |
---|---|---|---|
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
|
|
| |
Required for all services |
|
|
...
Authentication method | Tenant ID | Client ID | Client secret | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
...
Rw ui tabs macro | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector:
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in
Editing the config.yaml file
Replace the placeholders with your required values following the description table below:
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Use the following command to add the Docker image to the system:
The Docker image can be deployed on the following services: DockerExecute the following command on the root directory
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the
To run the container using docker-compose, execute the following command from the
We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer:
Editing the JSON configuration
Replace the placeholders with your required values following the description table below: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Parameter | Data type | Type | Value range / Format | Details | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
| If the value is | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum length: | Use this param to give a unique id to this collector. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum length: | Use this param to give a valid name to this collector, like | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
| Use this param to identify the Devo Cloud where the events will be sent. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum length: | Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum length: | Use this param to identify the | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum length: | Use this param to identify the | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum length: | Use this param to give a unique id to this input service.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
| Use this param to enable or disable the given input logic when running the collector. If the value is | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum value: | Customize the maximum number of API requests per second. If not used, the default setting will be used:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| UUID format | This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| UUID format | This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Any non-whitespace character | This is the Client’s secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Minimum value: | The amount (in seconds) in which the service’s collection is scheduled.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| For the secure_scores service: For the rest of the services: | This will allow you to start from a specific date in case you want to ingest historic events. If not set, it will start at the current time.
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
| Only accepts | The
The detailed table destination depending on the
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collector Docker image | SHA-256 hash | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Code Block |
---|
gunzip -c <collector-ms-graph-collector-if-docker-image>-<1.2.0>.tgz | docker load |
Note |
---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace |
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Code Block |
---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
Note |
---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/
directory.
Code Block |
---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/
directory:
Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace |
Rw tab | ||
---|---|---|
|
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Alerts
Alerts are potential security issues within a customer's tenant that Microsoft or partner security solutions have identified and flagged for action or notification. With the Microsoft Graph alerts entity, you can unify and streamline the management of security issues across all integrated solutions.
Alerts Security Providers:
Microsoft Defender for Cloud
Azure Active Directory Identity Protection
Microsoft Defender for Cloud Apps
Microsoft Defender for Endpoint
Microsoft Defender for Identity
Microsoft 365
Azure Information Protection
Azure Sentinel
Secure Scores
Microsoft Secure Score is a security analytics solution that gives you visibility into your security portfolio and how to improve it. With a single score, you can better understand what you have done to reduce your risk in Microsoft solutions. You can also compare your score with other organizations and see how your score has been trending over time. The Microsoft Graph secureScore
and secureScoreControlProfile
entities help you balance your organization's security and productivity needs while enabling the appropriate mix of security features. You can also project what your score would be after you adopt security features.
Azure Active Directory reports (Sign-in, Audit, Provisioning)
Azure Active Directory (Azure AD) reports providing a comprehensive view of activity in your environment. The provided data enables you to:
Determine how your apps and services are utilized by your users.
Detect potential risks affecting the health of your environment.
Troubleshoot issues preventing your users from getting their work done.
These reports help you understand the behavior of users in your organization. There are three types of reports that this collector pulls from Azure AD:
Sign-ins: Information about sign-ins and how your resources are used by your users.
Audit: Information about changes applied to your tenants such as users and group management or updates applied to your tenant’s resources.
Provisioning: Activities performed by the provisioning service, such as the creation of a group in ServiceNow or a user imported from Workday.
Devo categorization and destination
Here you can see how each service that has configurable tagging will tag its events depending on the value of the tag_version
parameter:
...
Service
...
Old tagging (v1
)
...
New tagging (v2
)
...
secure_scores
...
cloud.msgraph.security.secure_scores.1
...
cloud.office365.security.scores.1.msgraph
...
secure_score_control_profiles
...
cloud.msgraph.security.secure_score_control_profiles.1
...
cloud.office365.security.scorecontrol.1.msgraph
...
alerts
...
cloud.msgraph.security.alerts.1
...
The tagging is based on the provider. See the next table.
The alerts
Service Now uses dynamic tagging based on the event’s provider field. This is the provider/tag correspondence:
...
Provider
...
New tagging (v2
)
...
IPC
...
cloud.azure.ad.alerts.1.msgraph
...
MCAS
...
cloud.office365.cloud_apps.alerts.1.msgraph
...
Microsoft Defender ATP
...
cloud.office365.security.alerts.1.msgraph
...
Office 365 Security and Compliance
...
cloud.azure.sentinel.alerts.1.msgraph
...
Azure Sentinel
...
cloud.office365.identity.alerts.1.msgraph
...
Azure Advanced Threat Protection
...
cloud.azure.securitycenter.alerts.1.msgraph
Info |
---|
If the event comes with a provider that is not present in the table above, the tagging will fall back to the old one ( |
Events service
...
title | Verify data collection |
---|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
...
Component
...
Description
...
Setup
...
The setup module is in charge of authenticating the service and managing the token expiration when needed.
...
Puller
...
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block |
---|
INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Access Token has been validated successfully
INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Setup for module <MicrosoftGraphNonTimeBasedPuller> has been successfully executed |
Puller output
A successful initial run has the following output messages for the puller module:
Info |
---|
Note that the |
Code Block |
---|
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 2 requests, retrieved all vendor list, detected 1 unique vendors
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> MicrosoftGraphNonTimeBasedVendorPuller(microsoft_graph#1,secure_score_control_profile#predefined,MicrosoftGraphNonTimeBasedPuller#SecureScore.None) -> Starting thread
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Data collection completed. Elapsed time: 4.708 seconds. Waiting for 55.292 second(s) until the next one |
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block |
---|
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms |
Info |
---|
The value |
Expand | ||
---|---|---|
| ||
his collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can only do that by re-creating a new collector instance from scratch since this collector does not implement a state restart mechanism. |
...
title | Troubleshooting |
---|
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Common for all services
...
Error type
...
Error ID
...
Error message
...
Cause
...
Solution
...
MicrosoftGraphPullerCredentialsException
...
1
...
Invalid tenant. This may happen if there are no active subscriptions for the tenant
...
The tenant is not valid according to Microsoft AD. Probably, the setup has not been correctly followed.
...
Revisit the setup instructions and check that all the steps have been correctly followed.
...
1
...
Access Token validation has been failed, code: {status_code}, text: {text_str}
...
There was a problem obtaining the authentication token.
...
Read the exact error to understand what is the real cause.
...
2
...
Access Token not valid or client_id does not exist
...
There was an unknown problem during the authentication..
...
Check that the credentials have been correctly set up.
...
MicrosoftGraphPullerCreationException
ModuleDefinitionError
...
1
...
"module_properties" property in service definition must exists / "module_properties" mandatory property is missing or empty
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
2
...
"module_properties" property in service definition must be a dictionary
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
3
...
"resource_type" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
4
...
"resource_type" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
5
...
"base_url_main" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
6
...
"base_url_main" property in service definition must be a string.
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
7
...
"base_url_vendor" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
8
...
"base_url_vendor" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
9
...
"tag_base" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
10
...
"tag_base" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
11
...
"http_status_valid_codes" property in service definition must exists.
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
12
...
"http_status_valid_codes" property in service definition must be a list
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
13
...
"login_url" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
14
...
"login_url" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
15
...
"scope" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
16
...
"scope" property in service definition must be a string.
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
MicrosoftGraphPullerCreationException
...
37
...
"tag_version" property for {service_name} from configuration file is not valid, expected: v1, v2 or None
...
This configuration parameter expects a string with value v1, v2 or null but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string with value v1, v2 or null.
...
MicrosoftGraphPullerRetrieveException
...
1
...
Not defined "next_page_url"
...
When retrieving data from the MS Graph API, we expect that each paginated response has a next_page_url
property in its JSON. This error occurs when this property was not found.
...
Enable debug mode and inspect the requested URLs. Try to replicate those to see the response obtained.
...
2
...
Response text: "{response_text}", HTTP error: {http_error}
...
When retrieving data from the MS Graph API, we expect that the response code is within the [200-400) range; otherwise (HTTP response code ≥ 400), it will raise this error.
...
The error will include the response’s text. This should tell you what the problem is. The solution will depend on the type of the problem.
...
MicrosoftGraphPullerConnectionException
...
3
...
Operation timed out: {error_message}
...
When retrieving data from the MS Graph API, the server took too long to respond and the connection was closed.
...
Check your network connection and that the MS Graph API is operative.
...
4
...
Connection error detected: {error_message}
...
Some other error regarding the connection with the MS Graph API server occurred.
...
You should read the actual error message to understand the underlying issue and know how to solve it.
...
MicrosoftGraphPullerRetrieveException
...
5
...
{error_message}
...
Some unknown error occurred.
...
You should read the actual error message to understand the underlying issue and know how to solve it.
Non-time-base services (secure_score_control_profiles
)
...
Error type
...
Error ID
...
Error message
...
Cause
...
Solution
...
MicrosoftGraphPullerCreationException
...
17
...
"request_max_items_per_request" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
18
...
"request_max_items_per_request" property in service definition must be an intege
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
19
...
"requests_per_minute" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
20
...
"requests_per_minute" property in service definition must be an integer
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
21
...
"requests_per_minute" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
22
...
"requests_per_minute" property in service definition must be an integer
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
23
...
"new_service_name" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
27
...
"tag" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
28
...
"credentials" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
29
...
"credentials" property in configuration must be a dictionary
...
This configuration parameter expects a JSON object value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a JSON object.
...
30
...
"tenant_id" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
31
...
"tenant_id" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
32
...
"client_id" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
33
...
"client_id" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
34
...
"client_secret" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
35
...
"client_secret" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
Time-based (alerts
, secure_scores
) & audit (audit
, provisioning
, signIn
, signIn_nonInteractive
, signIn_servicePrincipal
, signIn_managedIdentity
) services
...
Error Type
...
Error ID
...
Error Message
...
Cause
...
Solution
...
ModuleDefinitionError
...
5
...
"base_url_main_only_first_page" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
6
...
"base_url_main_only_first_page" property in service definition must be a boolean
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
7
...
"base_url_vendor_with_sub_provider" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
8
...
"base_url_vendor_with_sub_provider" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
17
...
"base_url_main_items_per_request" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
18
...
"base_url_main_items_per_request" property in service definition must be an integer
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
19
...
"base_url_main_items_per_request" property in service definition must be a positive value
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
17
...
"base_url_vendor_items_per_request" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
18
...
"base_url_vendor_items_per_request" property in service definition must be an integer
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
19
...
"base_url_vendor_items_per_request" property in service definition must be a positive value
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
20
...
"max_result_set_size" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
21
...
"max_result_set_size" property in service definition must be an integer
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
22
...
"max_result_set_size" property in service definition must be a positive value
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
24
...
"legacy_provider_mapping_old_new" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
25
...
"requests_per_minute" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
26
...
"requests_per_minute" property in service definition must be an integer
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
27
...
"requests_per_minute" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
28
...
"requests_per_minute" property in service definition must be an integer
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
29
...
"timestamp_field" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
30
...
"timestamp_field" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
31
...
"start_time_regex" property in service definition must exists
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
32
...
"start_time_regex" property in service definition must be a string
...
This a programming error that we should not except to happen.
...
Contact Devo Support.
...
InputConfigurationError
...
1
...
"microsoft_graph" mandatory property is missing or empty
...
The input configuration is missing.
...
Ensure that the configuration includes an input configuration.
...
2
...
"microsoft_graph" property must be a dictionary
...
The input configuration expects to have a JSON object value, but it has parsed a different type of value.
...
Ensure that the configuration for this input is a JSON object.
...
3
...
"credentials" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
4
...
"credentials" property in configuration must be a dictionary
...
This configuration parameter expects a JSON object value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a JSON object.
...
5
...
"tenant_id" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
6
...
"tenant_id" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
7
...
"client_id" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
8
...
"client_id" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
9
...
"client_secret" property in configuration must exists
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
10
...
"client_secret" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
ServiceConfigurationError
...
1
...
"{service_name}" mandatory property is missing or empty
...
This configuration parameter is missing.
...
Ensure that this parameter is present and has a value.
...
ServiceConfigurationError
...
2
...
"{service_name}" property must be a dictionary
...
This configuration parameter expects a JSON object value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a JSON object.
...
MicrosoftGraphPullerCreationException
...
27
...
"tag" property in configuration must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
36
...
"start_time" property in service definition must be a string
...
This configuration parameter expects a string value, but it has parsed a different type of value.
...
Ensure that the value for this parameter is a string.
...
37
...
"start_time" property from configuration file format is not valid, expected: "{start_time_regex}"
...
This configuration parameter expects a date that matches the indicated regular expression, but it did not match.
...
Ensure that the value for this parameter is a valid date according to the indicated regular expression.
...
38
...
"tag_version" property for {self.service_name} from configuration file is not valid, received "v2", but there is no "tag_base_v2" property in module definitions
...
You set the tag_version
parameter to v2
, but the service does not have a v2
tagging.
...
Contact Devo Support.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
...
title | Verify collector operations |
---|
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
Code Block |
---|
INFO MainProcess::MainThread -> Added "[collector_dir]" directory to the Python path
INFO MainProcess::MainThread -> Added "[collector_dir]/config_internal" directory to the Python path
INFO MainProcess::MainThread -> {"production_mode": false, "python_version": "3.9.12 (main, Apr 6 2022, 10:19:29) \n[Clang 12.0.5 (clang-1205.0.22.9)]", "current_directory": "[collector_dir]", "exists_config_dir": true, "exists_config_internal_dir": true, "exists_certs_dir": true, "exists_credentials_dir": true}
INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-test-local.yaml", "job_config_loc": null, "collector_config_loc": null}
INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
INFO MainProcess::MainThread -> "/etc/devo/job" does not exists
INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
INFO MainProcess::MainThread -> "/etc/devo/collector" does not exists
INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "[collector_dir]/config/config-test-local.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
INFO MainProcess::MainThread -> {"build_time": "UNKNOWN", "os_info": "macOS-12.5.1-x86_64-i386-64bit", "collector_name": "microsoft_graph", "collector_version": "1.2.0", "collector_owner": "integrations_factory@devo.com", "started_at": "2022-09-01T14:57:53.453415Z"}
INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 10000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-us.devo.io', 'port': 443, 'type': 'SSL', 'chain': 'chain.cer', 'cert': 'cert.cer', 'key': 'key.key', 'concurrent_connections': 1, 'period_sender_stats_in_seconds': 300, 'activate_final_queue': False, 'threshold_for_using_gzip_in_transport_layer': 1.1, 'compression_level': 6, 'compression_buffer_in_bytes': 51200, 'generate_metrics': False}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
INFO InputProcess::MainThread -> Process Started
INFO OutputProcess::MainThread -> Process started
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> [GC] global: 67.8% -> 67.8%, process: RSS(38.10MiB -> 38.16MiB), VMS(32.82GiB -> 32.82GiB)
INFO InputProcess::MainThread -> InputThread(microsoft_graph,1) - Starting thread (execution_period=600s)
INFO InputProcess::MainThread -> ServiceThread(microsoft_graph,1,secure_score_control_profile,predefined) - Starting thread (execution_period=600s)
INFO InputProcess::MainThread -> MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Starting thread
INFO InputProcess::MainThread -> MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) - Starting thread
WARNING InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Waiting until setup will be executed
INFO InputProcess::MainThread -> [GC] global: 67.8% -> 67.8%, process: RSS(38.35MiB -> 38.35MiB), VMS(32.70GiB -> 32.70GiB)
INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-us.devo.io:443", "chain_path": "[collector_dir]/certs/chain.cer", "cert_path": "[collector_dir]/certs/cert.cer", "key_path": "[collector_dir]/certs/key.key", "transport_layer_type": "SSL"}, hostname: "2019EMEA0386.local", session_id: "4532931840" |
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
Code Block |
---|
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds) |
Info |
---|
By default, these information traces will be displayed every 10 minutes. |
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal
, standard
, and lookup
). This collector uses the following Sender Services:
...
Sender services
...
Description
...
internal_senders
...
In charge of delivering internal metrics to Devo such as logging traces or metrics.
...
standard_senders
...
In charge of delivering pulled events to Devo.
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
...
Logging trace
...
Description
...
Number of available senders: 1
...
Displays the number of concurrent senders available for the given Sender Service.
...
sender manager internal queue size: 0
...
Displays the items available in the internal sender queue.
Info |
---|
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
...
Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)
...
Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
44 events were sent to Devo since the collector started.
The last checkpoint timestamp was
2022-06-28 10:39:22.511671+00:00
.21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required
0.007 seconds
to be delivered.
Info |
---|
By default these traces will be shown every 10 minutes. |
Expand | ||||
---|---|---|---|---|
| ||||
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
|
Expand | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
|
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll directory audit records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll provisioning audit records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll sign-in records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||
Internal process and deduplication methodAll sign-in_v2 records are continuously pulled subject to the Devo categorization and destinationEvents of this service are ingested into the following tables:
|
Expand | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||
Internal process and deduplication methodAll alert records are continuously pulled subject to the Devo categorization and destinationEvents of this service are ingested into the following tables:
|
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll alerts_v2 records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll secure score records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Expand | ||
---|---|---|
| ||
Internal process and deduplication methodAll secure score profile records are continuously pulled subject to the Devo categorization and destinationAll events of this service are ingested into the table |
Restart the persistence for a service
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc
parameter to a different one.Save the changes.
Restart the collector. The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Common logic
Error type | Error ID | Error message | Cause | Solution |
---|---|---|---|---|
| 1 | Invalid | The configured | Update the |
| 2 | Invalid | The configured | Update the |
| 101 | Failed to fetch OAuth token from | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
| 102 | Failed to fetch data from | The provided credentials, base URL, and/or token endpoint are incorrect. | Revisit the configuration steps and ensure that the correct values were specified in the config file. |
| 401 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 401 response. | Ensure that the provided credentials are correct and provide read access to the targeted data. |
| 429 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 429 response. | The collector will attempt to retry requests (default up to 3 times) and |
| 498 | Error during API call to [API provider HTML error response here] | The server returned an HTTP 500 response. | If the API returns a 500 but successfully completes subsequent runs then you may ignore this error. If the API repeatedly returns a 500 error, ensure the server is reachable and operational. |
Collector operations
Verify collector operations
Expand | ||
---|---|---|
| ||
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration. A successful run has the following output messages for the initializer module:
|
Expand | ||
---|---|---|
| ||
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module:
|
Expand | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
The Integrations Factory Collector SDK has 3 different sender services depending on the event type to deliver (internal, standard, and lookup). This collector uses the following Sender Services:
|
Expand | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
|
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
Code Block |
---|
INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB ->
410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB ->
705.28MiB)Change log |
Anchor | ||||
---|---|---|---|---|
|
1.X to 2.X migration guide
Version 2.0.0 introduced several changes to the collector's configuration. When upgrading the collector, users must make changes to their configuration file to ensure that the collector continues to work as expected.
Note |
---|
Important Note that |
In versions 1.X of the collector, some services had a config parameter tag_version
with values v1
or v2
. The effect of this parameter is that the destination table for these services will be different. These are the destination tables according to the tag_version
value:
v1 | v2 (default) |
---|---|
|
|
|
|
In the new collector 2.0.0, the old config parameter tag_version
has been removed. The same effect as v1
can be made using override_tag
, with these values:
Service |
|
---|---|
|
|
|
|
For instance, if we have this in the old config:
Code Block |
---|
secure_scores:
tag_version: v1
secure_score_control_profile:
tag_version: v1 |
you should change it to:
Code Block |
---|
secure_scores:
override_tag: cloud.msgraph.security.scores.2
secure_score_control_profile:
override_tag: cloud.msgraph.security.scorecontrol.2 |
Note |
---|
Important Note that security alerts are sent to different tables according to their categories. |
In old versions of this collector, all alerts were sent to the table cloud.msgraph.security.alerts
However, as there are alerts of several types, in version 2.0.0 alerts are categorized according to their vendorInformation-provider
field (their vendor) and sent to different tables:
Vendor | Old table | New table |
---|---|---|
IPC |
|
|
MCAS |
| |
Microsoft Defender ATP |
| |
Microsoft 365 Defender |
| |
Office 365 Security and Compliance |
| |
Azure Sentinel |
| |
ASC |
| |
Azure Advanced Threat Protection |
| |
Others |
|
It is possible to avoid this feature and send all alerts to the same table by editing the config file and changing the tag:
Code Block |
---|
alerts:
override_tag: cloud.msgraph.security.alerts.2 |
Config file comparison
Below you will find a sample 1.7.1 configuration file and a sample 2.0.0 configuration file.
Expand | ||
---|---|---|
| ||
|
Expand | ||
---|---|---|
| ||
|
Config file changes
The URL endpoints (
override_base_url_main
,override_base_url_vendor
,override_base_url_vendor_with_sub_provider
,override_login_url
) have been moved from the individual services to the global configuration section.override_base_url_main
has been renamed tooverride_base_url
.tag_version
has been removed.pull_sliding_window_timespan_period
has been removed.reset_persistence_auth
has been removed.override_time_delta_in_days
has been removed.ms_365_environment
has been replaced byoverride_top_level_domain
. GCC High Gov should useus
in theoverride_top_level_domain
. Alternatively, users can useoverride_base_url
to specify the GCC High Gov base URL.additional_filter
has been added to all services. Users can use this field to specify additional filters that will be applied when querying the Microsoft Graph API.The collector can use new services from Graph (beta endpoint in Graph), that services, that use to be in a separate service for each type, have been consolidated into one service called
signIns_v2
. Users should remove all three services from their config and use only thesignIns_v2
service.start_time
has been renamed tostart_time_in_utc
.
Persistence changes
The persistence object consists of the following fields:
persistence_version
,last_event_time_in_utc
,last_ids
, andnext_link
.The collector will automatically map old key names (e.g.
last_polled_timestamp
→last_event_time_in_utc
) to the appropriate value.
Change log
Release | Released on | Release type | Details | Recommendations | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| Improvements
Bug Fixing
|
| |||||||||||||||||||
|
| Improvements
|
| |||||||||||||||||||
|
| New features
Improvements
Bug Fixingfixing
|
| |||||||||||||||||||
|
|
| New features:
Improvements:
Bug Fixingfixing:
|
| ||||||||||||||||||
|
| Improvements:
Bug Fixingfixing:
|
| |||||||||||||||||||
|
| New features:
Improvements:
|
| |||||||||||||||||||
|
| Fixed bugs:
|
| |||||||||||||||||||
|
| Fixed bugs:
|
| |||||||||||||||||||
|
| Improvements:
|
| |||||||||||||||||||
|
| Improvements:
Bug fixing:
|
| |||||||||||||||||||
|
| New features:
Improvements:
|
|