Microsoft Graph collector

[ 1 Configuration requirements ] [ 2 Overview ] [ 3 Devo collector features ] [ 4 Data sources ] [ 5 Vendor setup ] [ 6 Minimum configuration required for basic pulling ] [ 7 Accepted authentication methods ] [ 8 Run the collector ] [ 9 Collector services detail ] [ 10 Collector operations ] [ 11 Change log for v1.x.x ]

Configuration requirements

To run this collector, there are some configurations detailed below that you need to consider.

Configuration	Details

Configuration	Details
Azure account	Azure account with admin level permissions and Azure AD tenant.
Credentials	The credentials configuration block has been filled correctly.

More information

Refer to the Vendor setup section to know more about these configurations.

Overview

The Microsoft Graph Collector provides the ability to collect data and intelligence from services such as Microsoft 365, Windows, and Enterprise Mobility and Security. This data collector is able to ingest security alerts, scores, provisioning, audit, and sign-ins retrieved from Microsoft products, allowing you to empower streamlined security operations and better defend against threats faced in Azure AD and Microsoft 365 environments.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`Allowed`
Running environments	`Collector server` `On-premise`
Populated Devo events	`Table`
Flattening preprocessing	`No`

Data sources

Data source	Description	API endpoint	Collector service name	Devo table	Available from release
Alerts	Represents potential security issues within a customer’s tenant that Microsoft or partner security solutions have identified. Refer to Microsoft documentation about Alert Resource Type for more information.	`https://graph.microsoft.com/v1.0/security/alerts?$count=true&$filter=eventDateTime+ge+{start_time}+AND+vendorInformation/provider+eq+'{provider}'&$orderby=eventDateTime+asc&$top={items_per_vendor_request}`	`alerts`	Starting from v1.2.0, the destination table depends on the tag_version configuration parameter: `v1`: `ºv2`: the destination will be dynamic, depending on the provider: IPC: `cloud.azure.ad.alerts`. MCAS: `cloud.office365.cloud_apps.alerts`. Microsoft Defender ATP: `cloud.office365.endpoint.alerts`. Office 365 Security and Compliance: `cloud.office365.security.alerts`. Azure Sentinel: `cloud.azure.sentinel.alerts`. ASC: `cloud.office365.identity.alerts`. Azure Advanced Threat Protection: `cloud.azure.securitycenter.alerts`. For any other provider, it will fall back in `v1`'s tagging. The detailed table destination will depend on the tag_version’s value `v1.0.0`available in the Devo categorization and destination section.	`v1.0.0`
Secure scores	Represents a tenant's secure score per day of scoring data, at the tenant and control level. By default, 90 days of data is held. Refer to the Microsoft documentation for more information about Secure scores resources types.	`https://graph.microsoft.com/v1.0/security/secureScores?$count=true&$filter=createdDateTime+ge+{start_time}+AND+vendorInformation/provider+eq+'{provider}'&$orderby=createdDateTime+asc,vendorInformation/provider+asc&$top={items_per_vendor_request}`	`secure_scores`	Starting from v1.2.0, the destination table depends on the `tag_version` configuration parameter: `v1`: `cloud.msgraph.security.secure_scores` `v2`: `cloud.office365.security.scores` The detailed table destination depending on the tag_version's value is available in the Devo categorization and destination section.	`v1.0.0`
Secure score control profiles	Represents a tenant's secure score per control data. Refer to the Microsoft documentation for more information about Secure score control profiles.	`https://graph.microsoft.com/v1.0/security/secureScoreControlProfiles?$count=true`	`secure_score_control_profiles`	Starting from v1.2.0, the destination table depends on the `tag_version` configuration parameter: `v1`: `cloud.msgraph.security.secure_score_control_profiles` `v2`: `cloud.office365.security.scorecontrol`	`v1.0.0`
Directory audit	Represents the directory audit items and its collection. Refer to the Microsoft documentation for more information about Directory audit.	`https://graph.microsoft.com/v1.0/auditLogs/directoryAudits?$filter=activityDateTime ge {start_time}&$orderby=activityDateTime+asc&$top={items_per_main_request}`	`audit`	`cloud.azure.ad.audit`	`v1.2.0`
Provisioning	Represents an action performed by the Azure AD Provisioning service and its associated properties. Refer to the Microsoft documentation for more information about Provisioning.	`https://graph.microsoft.com/beta/auditLogs/provisioning?$filter=activityDateTime ge {start_time}&$orderby=activityDateTime+asc&$top={items_per_main_request}`	`provisioning`	`cloud.azure.ad.audit`	`v1.2.0`
Sign-in	Details user and application sign-in activity for a tenant (directory). Refer to the Microsoft documentation for more information about Sign-in.	`signIn`: `https://graph.microsoft.com/v1.0/auditLogs/signIns?$orderby=createdDateTime+asc&$top={items_per_main_request}` `signIn_nonInteractive`: `https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'nonInteractiveUser')&$orderby=createdDateTime+asc&$top={items_per_main_request}` `signIn_servicePrincipal`: `https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'servicePrincipal')&$orderby=createdDateTime+asc&$top={items_per_main_request}` `signIn_managedIdentity`: `https://graph.microsoft.com/beta/auditLogs/signIns?&$filter=signInEventTypes/any(x:x eq 'managedIdentity')&$orderby=createdDateTime+asc&$top={items_per_main_request}`	`signIn` `signIn_nonInteractive` `signIn_servicePrincipal` `signIn_managedIdentity`	`signIn`: `cloud.azure.ad.signin` `signIn_nonInteractive`: `cloud.azure.ad.noninteractive_user_signin` `signIn_servicePrincipal`: `cloud.azure.ad.service_principal_signin` `signIn_managedIdentity`: `cloud.azure.ad.managed_identity_signin`	`v1.2.0`

Vendor setup

Microsoft Graph data collector works over Microsoft products. To activate the resources from the Microsoft Graph API, you need:

An Azure account that has an active subscription.
The Azure account must have permission to manage applications in Azure Active Directory (Azure AD).
A working Azure AD tenant.

You will need to register a new application and apply the required permissions to the corresponding resources to authenticate the collector in order to retrieve the data.

	Action	Steps
1	Register and configure the application	Go to Azure portal and click on Azure Active Directory. Click on App registration on the left-menu side. Then click on + New registration. On the Register and Application page: Name the application. Select Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox) in Supported Accounts type. In Redirect URI (optional) leave it as default (blank). Click Register. App registration page will open. Click on your app to configure it and give it permissions. You will see your app’s dashboard with information (docs, endpoints, etc.) when clicking it. Click Authentication on the left-menu side, then choose + Add a platform and select Mobile and desktop application. Select the three redirects URIs: https://login.microsoftonline.com/common/oauth2/nativeclient https://login.live.com/oauth20_desktop.srf msale36f3a02-3eef-437b-874e-8a0aa29a2bf0://auth Click Configure.
2	Grant the required permissions	Go to API permissions on the left-menu side. Click + Add permission in case you don’t have Microsoft Graph in the API/Permission list. Select Application permissions and search for Security. Check `SecurityEvents.Read.All`. Repeat the same step 3 for `AuditLog.Read.All`, `Directory.Read.All` and `User.Read`. If you did everything correctly, permissions will display. Select Grant admin consent for the applications.
3	Obtain the requires credentials for the collector	Go to Certificates & Secrets, select + New client secret . Named it and copy the token value. Go to Overview to get your Tenant ID and Client ID and copy both values.

Permission reference per service

Collector service	Resource	Required permissions	Microsoft documentation
`alerts`	alerts	SecurityEvents.Read.All	List alerts
`secure_scores`	secureScores	SecurityEvents.Read.All	List secureScores
`secure_score_control_profiles`	secureScoreControlProfiles	SecurityEvents.Read.All	List secureScoreControlProfiles
`audit`	directoryAudits	AuditLog.Read.All and Directory.Read.All	List directoryAudits
`provisioning`	provisioningObjectSummary	AuditLog.Read.All and Directory.Read.All	List provisioningObjectSummary
`signIn`	signIns	AuditLog.Read.All and Directory.Read.All	List signIns
Required for all services	authentication	User.Read	Microsoft Graph permissions

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

Setting	Details
`tenant_id_value`	This is the Tenant’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.
`client_id_value`	This is the Client’s ID you created in Azure AD. You can obtain it from the Overview page in your registered application.
`client_secret_value`	This is the Client’s secret you created in Azure AD. You can obtain it from the Certificates & secrets page in your registered application.

Accepted authentication methods

This collector only accepts one single authentication method. You will have to fill the following properties on the credentials configuration block:

Authentication method	Tenant ID	Client ID	Client secret
OAuth2 Client credentials	REQUIRED	REQUIRED	REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Alerts

Alerts are potential security issues within a customer's tenant that Microsoft or partner security solutions have identified and flagged for action or notification. With the Microsoft Graph alerts entity, you can unify and streamline the management of security issues across all integrated solutions.

Alerts Security Providers:

Microsoft Defender for Cloud
Azure Active Directory Identity Protection
Microsoft Defender for Cloud Apps
Microsoft Defender for Endpoint
Microsoft Defender for Identity
Microsoft 365
Azure Information Protection
Azure Sentinel

Secure Scores

Microsoft Secure Score is a security analytics solution that gives you visibility into your security portfolio and how to improve it. With a single score, you can better understand what you have done to reduce your risk in Microsoft solutions. You can also compare your score with other organizations and see how your score has been trending over time. The Microsoft Graph secureScore and secureScoreControlProfile entities help you balance your organization's security and productivity needs while enabling the appropriate mix of security features. You can also project what your score would be after you adopt security features.

Azure Active Directory reports (Sign-in, Audit, Provisioning)

Azure Active Directory (Azure AD) reports providing a comprehensive view of activity in your environment. The provided data enables you to:

Determine how your apps and services are utilized by your users.
Detect potential risks affecting the health of your environment.
Troubleshoot issues preventing your users from getting their work done.

These reports help you understand the behavior of users in your organization. There are three types of reports that this collector pulls from Azure AD:

Sign-ins: Information about sign-ins and how your resources are used by your users.
Audit: Information about changes applied to your tenants such as users and group management or updates applied to your tenant’s resources.
Provisioning: Activities performed by the provisioning service, such as the creation of a group in ServiceNow or a user imported from Workday.

Devo categorization and destination

Here you can see how each service that has configurable tagging will tag its events depending on the value of the tag_version parameter:

Service	Old tagging (`v1`)	New tagging (`v2`)
`secure_scores`	`cloud.msgraph.security.secure_scores.1`	`cloud.office365.security.scores.1.msgraph`
`secure_score_control_profiles`	`cloud.msgraph.security.secure_score_control_profiles.1`	`cloud.office365.security.scorecontrol.1.msgraph`
`alerts`	`cloud.msgraph.security.alerts.1`	The tagging is based on the provider. See the next table.

The alerts Service Now uses dynamic tagging based on the event’s provider field. This is the provider/tag correspondence:

Provider	New tagging (`v2`)
IPC	`cloud.azure.ad.alerts.1.msgraph`
MCAS	`cloud.office365.cloud_apps.alerts.1.msgraph`
Microsoft Defender ATP	`cloud.office365.security.alerts.1.msgraph`
Office 365 Security and Compliance	`cloud.azure.sentinel.alerts.1.msgraph`
Azure Sentinel	`cloud.office365.identity.alerts.1.msgraph`
Azure Advanced Threat Protection	`cloud.azure.securitycenter.alerts.1.msgraph`

Events service

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Access Token has been validated successfully
INFO InputProcess::MicrosoftGraphPullerSetup(unknown,microsoft_graph#1,secure_score_control_profile#predefined) -> Setup for module <MicrosoftGraphNonTimeBasedPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 2 requests, retrieved all vendor list, detected 1 unique vendors
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> MicrosoftGraphNonTimeBasedVendorPuller(microsoft_graph#1,secure_score_control_profile#predefined,MicrosoftGraphNonTimeBasedPuller#SecureScore.None) -> Starting thread
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms
INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Data collection completed. Elapsed time: 4.708 seconds. Waiting for 55.292 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

INFO InputProcess::MicrosoftGraphNonTimeBasedPuller(microsoft_graph,1,secure_score_control_profile,predefined) -> Sent 4 requests, messages(received/sent): 274/274, avg_time_per_source_message: 17.177 ms

his collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can only do that by re-creating a new collector instance from scratch since this collector does not implement a state restart mechanism.

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Common for all services

Error type	Error ID	Error message	Cause	Solution

Error type	Error ID	Error message	Cause	Solution
`MicrosoftGraphPullerCredentialsException`	1	Invalid tenant. This may happen if there are no active subscriptions for the tenant	The tenant is not valid according to Microsoft AD. Probably, the setup has not been correctly followed.	Revisit the setup instructions and check that all the steps have been correctly followed.
	1	Access Token validation has been failed, code: {status_code}, text: {text_str}	There was a problem obtaining the authentication token.	Read the exact error to understand what is the real cause.
	2	Access Token not valid or client_id does not exist	There was an unknown problem during the authentication..	Check that the credentials have been correctly set up.
`MicrosoftGraphPullerCreationException` `ModuleDefinitionError`	1	"module_properties" property in service definition must exists / "module_properties" mandatory property is missing or empty	This a programming error that we should not except to happen.	Contact Devo Support.
	2	"module_properties" property in service definition must be a dictionary	This a programming error that we should not except to happen.	Contact Devo Support.
	3	"resource_type" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	4	"resource_type" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	5	"base_url_main" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	6	"base_url_main" property in service definition must be a string.	This a programming error that we should not except to happen.	Contact Devo Support.
	7	"base_url_vendor" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	8	"base_url_vendor" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	9	"tag_base" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	10	"tag_base" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	11	"http_status_valid_codes" property in service definition must exists.	This a programming error that we should not except to happen.	Contact Devo Support.
	12	"http_status_valid_codes" property in service definition must be a list	This a programming error that we should not except to happen.	Contact Devo Support.
	13	"login_url" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	14	"login_url" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	15	"scope" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	16	"scope" property in service definition must be a string.	This a programming error that we should not except to happen.	Contact Devo Support.
`MicrosoftGraphPullerCreationException`	37	"tag_version" property for {service_name} from configuration file is not valid, expected: v1, v2 or None	This configuration parameter expects a string with value v1, v2 or null but it has parsed a different type of value.	Ensure that the value for this parameter is a string with value v1, v2 or null.
`MicrosoftGraphPullerRetrieveException`	1	Not defined "next_page_url"	When retrieving data from the MS Graph API, we expect that each paginated response has a `next_page_url` property in its JSON. This error occurs when this property was not found.	Enable debug mode and inspect the requested URLs. Try to replicate those to see the response obtained.
`MicrosoftGraphPullerRetrieveException`	2	Response text: "{response_text}", HTTP error: {http_error}	When retrieving data from the MS Graph API, we expect that the response code is within the [200-400) range; otherwise (HTTP response code ≥ 400), it will raise this error.	The error will include the response’s text. This should tell you what the problem is. The solution will depend on the type of the problem.
`MicrosoftGraphPullerConnectionException`	3	Operation timed out: {error_message}	When retrieving data from the MS Graph API, the server took too long to respond and the connection was closed.	Check your network connection and that the MS Graph API is operative.
`MicrosoftGraphPullerConnectionException`	4	Connection error detected: {error_message}	Some other error regarding the connection with the MS Graph API server occurred.	You should read the actual error message to understand the underlying issue and know how to solve it.
`MicrosoftGraphPullerRetrieveException`	5	{error_message}	Some unknown error occurred.	You should read the actual error message to understand the underlying issue and know how to solve it.

Non-time-base services (`secure_score_control_profiles`)

Error type	Error ID	Error message	Cause	Solution

Error type	Error ID	Error message	Cause	Solution
`MicrosoftGraphPullerCreationException`	17	"request_max_items_per_request" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	18	"request_max_items_per_request" property in service definition must be an intege	This a programming error that we should not except to happen.	Contact Devo Support.
	19	"requests_per_minute" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	20	"requests_per_minute" property in service definition must be an integer	This a programming error that we should not except to happen.	Contact Devo Support.
	21	"requests_per_minute" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	22	"requests_per_minute" property in service definition must be an integer	This a programming error that we should not except to happen.	Contact Devo Support.
	23	"new_service_name" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	27	"tag" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
	28	"credentials" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	29	"credentials" property in configuration must be a dictionary	This configuration parameter expects a JSON object value, but it has parsed a different type of value.	Ensure that the value for this parameter is a JSON object.
	30	"tenant_id" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	31	"tenant_id" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
	32	"client_id" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	33	"client_id" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
	34	"client_secret" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	35	"client_secret" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.

Time-based (`alerts`, `secure_scores`) & audit (`audit`, `provisioning`, `signIn`, `signIn_nonInteractive`, `signIn_servicePrincipal`, `signIn_managedIdentity`) services

Error Type	Error ID	Error Message	Cause	Solution

Error Type	Error ID	Error Message	Cause	Solution
`ModuleDefinitionError`	5	"base_url_main_only_first_page" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	6	"base_url_main_only_first_page" property in service definition must be a boolean	This a programming error that we should not except to happen.	Contact Devo Support.
	7	"base_url_vendor_with_sub_provider" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	8	"base_url_vendor_with_sub_provider" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	17	"base_url_main_items_per_request" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	18	"base_url_main_items_per_request" property in service definition must be an integer	This a programming error that we should not except to happen.	Contact Devo Support.
	19	"base_url_main_items_per_request" property in service definition must be a positive value	This a programming error that we should not except to happen.	Contact Devo Support.
	17	"base_url_vendor_items_per_request" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	18	"base_url_vendor_items_per_request" property in service definition must be an integer	This a programming error that we should not except to happen.	Contact Devo Support.
	19	"base_url_vendor_items_per_request" property in service definition must be a positive value	This a programming error that we should not except to happen.	Contact Devo Support.
	20	"max_result_set_size" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	21	"max_result_set_size" property in service definition must be an integer	This a programming error that we should not except to happen.	Contact Devo Support.
	22	"max_result_set_size" property in service definition must be a positive value	This a programming error that we should not except to happen.	Contact Devo Support.
	24	"legacy_provider_mapping_old_new" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	25	"requests_per_minute" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	26	"requests_per_minute" property in service definition must be an integer	This a programming error that we should not except to happen.	Contact Devo Support.
	27	"requests_per_minute" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	28	"requests_per_minute" property in service definition must be an integer	This a programming error that we should not except to happen.	Contact Devo Support.
	29	"timestamp_field" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	30	"timestamp_field" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
	31	"start_time_regex" property in service definition must exists	This a programming error that we should not except to happen.	Contact Devo Support.
	32	"start_time_regex" property in service definition must be a string	This a programming error that we should not except to happen.	Contact Devo Support.
`InputConfigurationError`	1	"microsoft_graph" mandatory property is missing or empty	The input configuration is missing.	Ensure that the configuration includes an input configuration.
	2	"microsoft_graph" property must be a dictionary	The input configuration expects to have a JSON object value, but it has parsed a different type of value.	Ensure that the configuration for this input is a JSON object.
	3	"credentials" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	4	"credentials" property in configuration must be a dictionary	This configuration parameter expects a JSON object value, but it has parsed a different type of value.	Ensure that the value for this parameter is a JSON object.
	5	"tenant_id" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	6	"tenant_id" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
	7	"client_id" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	8	"client_id" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
	9	"client_secret" property in configuration must exists	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
	10	"client_secret" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
`ServiceConfigurationError`	1	"{service_name}" mandatory property is missing or empty	This configuration parameter is missing.	Ensure that this parameter is present and has a value.
`ServiceConfigurationError`	2	"{service_name}" property must be a dictionary	This configuration parameter expects a JSON object value, but it has parsed a different type of value.	Ensure that the value for this parameter is a JSON object.
`MicrosoftGraphPullerCreationException`	27	"tag" property in configuration must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
	36	"start_time" property in service definition must be a string	This configuration parameter expects a string value, but it has parsed a different type of value.	Ensure that the value for this parameter is a string.
	37	"start_time" property from configuration file format is not valid, expected: "{start_time_regex}"	This configuration parameter expects a date that matches the indicated regular expression, but it did not match.	Ensure that the value for this parameter is a valid date according to the indicated regular expression.
	38	"tag_version" property for {self.service_name} from configuration file is not valid, received "v2", but there is no "tag_base_v2" property in module definitions	You set the `tag_version` parameter to `v2`, but the service does not have a `v2` tagging.	Contact Devo Support.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services	Description

Sender services	Description
`internal_senders`	In charge of delivering internal metrics to Devo such as logging traces or metrics.
`standard_senders`	In charge of delivering pulled events to Devo.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace	Description

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

Change log for v1.x.x

Release	Released on	Release type	Details	Recommendations

Release	Released on	Release type	Details	Recommendations
`v1.2.0`	Aug 2, 2022	NEW FEATURE IMPROVEMENT	New features: New supported sources Sign In (`signIn` service) Audit (`audit` service) Provisioning (`provisioning` service) Previous services modification The new tagging introduced in the previous v1.1.3 release is now customizable through the `tag_version` service parameter. The default tagging has been reverted to the original one. The `alerts` source, when setting the `tag_version` to `v2`, will try to categorize the events by applying different tags based on the event’s provider. Improvements: Token validation is now performed against the corresponding endpoint.	`Upgrade`
`v1.3.0`	Nov 18, 2022	IMPROVEMENT BUG FIX	Improvements: `start_time` configuration parameter normalization for `audit` and `provisioning` services. Upgraded devocollectorsdk from 1.4.0 to 1.4.4b: Added: New "templates" functionality. New controlled stopping condition when any input thread fatally fails. Log traces for knowing the execution environment status (debug mode). Changed: Improved log trace details when runtime exceptions happen Refactored source code structure Fixes in the current puller template version The Docker container exits with the proper error code Bug fixing: Correct token validation when a Partial Content response is received. Use appropriate destination tag for provisioning events.	`Upgrade`
`v1.4.0`	Dec 2, 2022	IMPROVEMENT	Improvements: Automatic outdated `start_time` correction for audit-based services. New “reset persistence” functionality.	`Upgrade`
`v1.4.1`	Dec 2, 2022	BUG FIX	Fixed bugs: Fix error with vendor state when checking the `reset_persistence_auth` parameter. Allow using v2 tags for `secure_scores` and `secure_scores_control_profile` tags. Add missing Devo metadata into events.	`Upgrade`
`v1.4.2`	Dec 27, 2022	BUG FIX	Fixed bugs: Fixes bug with non-time-based puller state.	`Upgrade`
`v1.6.0`	Aug 2, 2022	NEW FEATURE	New features: The pulling mechanism now uses a sliding window to avoid event loss and duplication. Improvements: DevoCollectorSDK upgraded to v1.6.0: Added: More log traces related to execution environment details. Global rate limiters functionality. Extra checks for supporting MacOS as development environment. Obfuscation functionality. Changed: Some log traces now are shown less frequently. The default value for the logging frequency for "main" processes has been changed (to 120 seconds). Updated some Python Packages. Controlled stopping functionality more stable when using the "template". Improved some log messages related to Devo certificates (when using the Devo sender). Validate json objects before saving them to persistence (using filesystem). The v1.5.0 release has not been published.	`Recommended version`