If you are migrating from old 1.x.x versions, you can find a complete guide at Azure Collector Migration Guide.
Overview
Microsoft Azure is an ever-expanding set of cloud computing services to help your organization meet its business challenges. Azure gives you the freedom to build, manage, and deploy applications on a massive, global network using your preferred tools and frameworks.
Devo collector features
Features
Details
Allow parallel downloading (multipod)
Partial (supported for event_hubs services using Azure Blob Storage)
Running environments
collector server
on-premise
Populated Devo events
table
Flattening pre-processing
no
Allowed source events obfuscation
yes
Data source description
Data source
Description
API endpoint
Collector service name
Devo table
VM Metrics
With the advantages of the Microsoft Azure API, one can obtain metrics about the deployed Virtual Machines, gathering them on our platform, making it easier to query and analyze in the Devo platform and Activeboards.
Several Microsoft Azure services can generate some type of execution information to be sent to an EventHub service. (see next section)
Azure Event Hubs SDK
event_hubs and event_hubs_autodiscover
<auto_tag_description>
Event hubs: Auto-categorization of Microsoft Azure service messages
Many of the available Microsoft Azure services can generate some type of execution information to be sent to an EventHub service. This type of data can be categorized as events or metrics. The events, in turn, can be from different subtypes: audits, status, logs, etc.
All such data will be gathered by Devo’s Microsoft Azure collector and sent to our platform, where message auto-categorization functionality is enabled for sending the messages to relevant Devo tables in an automatic way.
Although EventHub is the service used for centralizing Azure services' data, it also generates information that can be sent to itself.
In case the amount of egress data exceeds Throughput per Unit limits set by Azure (2 MB/s or 4096 events per second), it won’t be possible for Devo to continue reliable ingestion of data. You can monitor ingress/egress throughput in Azure Portal EventHub Namespace, and based on trends/alerts, you can add another EventHub to resolve this. To avoid this from happening in the first place, please follow scalability guidance provided by Microsoft in their technical documentation.
Auto-categorization of Microsoft Azure service messages
In the table below are listed the patterns that will be used for detecting the message type, the Provider, Service, and Category pattern values would be used to route the message to the proper Devo table.
Each message stored in an EventHub service is generated by one data Provider and also by one Service, and finally, it's also having a Category field, which all together determine the message type.
Over time, the auto-categorization patterns have been improved and expanded in the different collector versions, the tables below contain the pattern values released in each version.
The collector versions not mentioned here are having changes not related to event mapping for auto-categorization functionality.
If none of the previous patterns are matched, the following ones will be applied:
Provider
Service
Category
Devo table
*
*
Administrative
cloud.azure.others.administrative
Autoscale
cloud.azure.others.autoscale
Policy
cloud.azure.others.policy
Recommendation
cloud.azure.others.recommendation
ResourceHealth
cloud.azure.others.resourcehealth
The basic type detection will be applied for other values:
Message type
Devo table
event
cloud.azure.others.events
metric
cloud.azure.eh.metrics
Version 1.0.24
Provider
Service
Category
Devo table
Since version
Microsoft.ContainerService
MANAGEDCLUSTERS
kube-audit
cloud.azure.aks.kube_audit
1.0.16
kube-audit-admin
cloud.azure.aks.kube_audit_admin
1.0.16
kube-controller-manager
cloud.azure.aks.kube_controller_manager
1.0.16
kube-scheduler
cloud.azure.aks.kube_scheduler
1.0.16
cluster-autoscaler
cloud.azure.aks.cluster_autoscaler
1.0.16
guard
cloud.azure.aks.guard
1.0.16
Policy
cloud.azure.aks.policy
1.0.16
Administrative
cloud.azure.aks.administrative
1.0.16
Microsoft.Network
APPLICATIONGATEWAYS
ApplicationGatewayAccessLog
cloud.azure.appgateway.access_log
1.0.16
ApplicationGatewayFirewallLog
cloud.azure.appgateway.firewall_log
1.0.16
Policy
cloud.azure.appgateway.policy
1.0.16
Administrative
cloud.azure.appgateway.administrative
1.0.16
AZUREFIREWALLS
AzureFirewallApplicationRule
cloud.azure.firewall.application_rule
1.0.16
AzureFirewallNetworkRule
cloud.azure.firewall.network_rule
1.0.16
AzureFirewallDnsProxy
cloud.azure.firewall.dns_proxy
1.0.16
FRONTDOORS
FrontdoorAccessLog
cloud.azure.frontdoor.access
1.0.24
FrontdoorWebApplicationFirewallLog
cloud.azure.frontdoor.waf
1.0.24
Microsoft.Storage
STORAGEACCOUNTS
Administrative
cloud.azure.storage.administrative
1.0.16
ResourceHealth
cloud.azure.storage.resourcehealth
1.0.16
Microsoft.Web
SITES
Administrative
cloud.azure.appservice.calculated_category
1.0.16
Policy
cloud.azure.appservice.policy
1.0.16
Microsoft.ContainerRegistry
REGISTRIES
ContainerRegistryLoginEvents
cloud.azure.contregistry.login
1.0.16
Microsoft.DBforPostgreSQL
SERVERS
PostgreSQLLogs
cloud.azure.postgresql.events
1.0.16
Microsoft.Compute
VIRTUALMACHINES
Administrative
cloud.azure.vm.administrative
1.0.16
ResourceHealth
cloud.azure.vm.resourcehealth
1.0.16
Policy
cloud.azure.vm.policy
1.0.16
Recommendation
cloud.azure.vm.recommendation
1.0.16
VIRTUALMACHINESCALESETS
Administrative
cloud.azure.vmscalesets.administrative
1.0.16
ResourceHealth
cloud.azure.vmscalesets.resourcehealth
1.0.16
Policy
cloud.azure.vmscalesets.policy
1.0.16
Autoscale
cloud.azure.vmscalesets.autoscale
1.0.16
Microsoft.DataFactory
FACTORIES
Administrative
cloud.azure.datafactory.administrative
1.0.16
Microsoft.Insights
ACTIVITYLOGALERTS
Alert
cloud.azure.monitor.alert
1.0.16
Microsoft.Security
LOCATIONS
Security
cloud.azure.securitycenter.security
1.0.16
Microsoft.KeyVault
VAULTS
AuditEvent
cloud.azure.keyvault.audit
1.0.16
Administrative
cloud.azure.keyvault.administrative
1.0.16
Policy
cloud.azure.keyvault.policy
1.0.16
Microsoft.aadiam
<empty>
SignInLogs
cloud.azure.ad.signin
1.0.16
AuditLogs
cloud.azure.ad.audit
1.0.16
NonInteractiveUserSignInLogs
cloud.azure.ad.noninteractive_user_signin
1.0.24 (overwrite previous)
ServicePrincipalSignInLogs
cloud.azure.ad.service_principal_signin
1.0.17
ProvisioningLogs
cloud.azure.ad.provisioning
1.0.17
ManagedIdentitySignInLogs
cloud.azure.ad.managed_identity_signin
1.0.24
Microsoft.OperationalInsights
WORKSPACES
Audit
cloud.azure.monitor.audit
1.0.17
MICROSOFT.SQL
SERVERS
AutomaticTuning
cloud.azure.sql.automatic_tuning
1.0.24
QueryStoreRuntimeStatistics
cloud.azure.sql.query_store_runtime
1.0.24
If none of the previous patterns are matched, the following ones will be applied:
Provider
Service
Category
Devo table
*
*
Administrative
cloud.azure.others.administrative
Autoscale
cloud.azure.others.autoscale
Policy
cloud.azure.others.policy
Recommendation
cloud.azure.others.recommendation
ResourceHealth
cloud.azure.others.resourcehealth
The basic type detection will be applied for other values:
Message type
Devo table
event
cloud.azure.eh.events
metric
cloud.azure.eh.metrics
Version 1.0.17
Provider
Service
Category
Devo table
Since version
Microsoft.ContainerService
MANAGEDCLUSTERS
kube-audit
cloud.azure.aks.kube_audit
1.0.16
kube-audit-admin
cloud.azure.aks.kube_audit_admin
1.0.16
kube-controller-manager
cloud.azure.aks.kube_controller_manager
1.0.16
kube-scheduler
cloud.azure.aks.kube_scheduler
1.0.16
cluster-autoscaler
cloud.azure.aks.cluster_autoscaler
1.0.16
guard
cloud.azure.aks.guard
1.0.16
Policy
cloud.azure.aks.policy
1.0.16
Administrative
cloud.azure.aks.administrative
1.0.16
Microsoft.Network
APPLICATIONGATEWAYS
ApplicationGatewayAccessLog
cloud.azure.appgateway.access_log
1.0.16
ApplicationGatewayFirewallLog
cloud.azure.appgateway.firewall_log
1.0.16
Policy
cloud.azure.appgateway.policy
1.0.16
Administrative
cloud.azure.appgateway.administrative
1.0.16
AZUREFIREWALLS
AzureFirewallApplicationRule
cloud.azure.firewall.application_rule
1.0.16
AzureFirewallNetworkRule
cloud.azure.firewall.network_rule
1.0.16
AzureFirewallDnsProxy
cloud.azure.firewall.dns_proxy
1.0.16
Microsoft.Storage
STORAGEACCOUNTS
Administrative
cloud.azure.storage.administrative
1.0.16
ResourceHealth
cloud.azure.storage.resourcehealth
1.0.16
Microsoft.Web
SITES
Administrative
cloud.azure.appservice.calculated_category
1.0.16
Policy
cloud.azure.appservice.policy
1.0.16
Microsoft.ContainerRegistry
REGISTRIES
ContainerRegistryLoginEvents
cloud.azure.contregistry.login
1.0.16
Microsoft.DBforPostgreSQL
SERVERS
PostgreSQLLogs
cloud.azure.postgresql.events
1.0.16
Microsoft.Compute
VIRTUALMACHINES
Administrative
cloud.azure.vm.administrative
1.0.16
ResourceHealth
cloud.azure.vm.resourcehealth
1.0.16
Policy
cloud.azure.vm.policy
1.0.16
Recommendation
cloud.azure.vm.recommendation
1.0.16
VIRTUALMACHINESCALESETS
Administrative
cloud.azure.vmscalesets.administrative
1.0.16
ResourceHealth
cloud.azure.vmscalesets.resourcehealth
1.0.16
Policy
cloud.azure.vmscalesets.policy
1.0.16
Autoscale
cloud.azure.vmscalesets.autoscale
1.0.16
Microsoft.DataFactory
FACTORIES
Administrative
cloud.azure.datafactory.administrative
1.0.16
Microsoft.Insights
ACTIVITYLOGALERTS
Alert
cloud.azure.monitor.alert
1.0.16
Microsoft.Security
LOCATIONS
Security
cloud.azure.securitycenter.security
1.0.16
Microsoft.KeyVault
VAULTS
AuditEvent
cloud.azure.keyvault.audit
1.0.16
Administrative
cloud.azure.keyvault.administrative
1.0.16
Policy
cloud.azure.keyvault.policy
1.0.16
Microsoft.aadiam
<empty>
SignInLogs
cloud.azure.ad.signin
1.0.16
AuditLogs
cloud.azure.ad.audit
1.0.16
NonInteractiveUserSignInLogs
cloud.azure.ad.signin
1.0.17
ServicePrincipalSignInLogs
cloud.azure.ad.service_principal_signin
1.0.17
ProvisioningLogs
cloud.azure.ad.provisioning
1.0.17
Microsoft.OperationalInsights
WORKSPACES
Audit
cloud.azure.monitor.audit
1.0.17
If none of the previous patterns are matched, the following ones will be applied:
Provider
Service
Category
Devo table
*
*
Administrative
cloud.azure.others.administrative
Autoscale
cloud.azure.others.autoscale
Policy
cloud.azure.others.policy
Recommendation
cloud.azure.others.recommendation
ResourceHealth
cloud.azure.others.resourcehealth
The basic type detection will be applied for other values:
Message type
Devo table
event
cloud.azure.eh.events
metric
cloud.azure.eh.metrics
Version 1.0.16
Provider
Service
Category
Devo table
Since version
Microsoft.ContainerService
MANAGEDCLUSTERS
kube-audit
cloud.azure.aks.kube_audit
1.0.16
kube-audit-admin
cloud.azure.aks.kube_audit_admin
1.0.16
kube-controller-manager
cloud.azure.aks.kube_controller_manager
1.0.16
kube-scheduler
cloud.azure.aks.kube_scheduler
1.0.16
cluster-autoscaler
cloud.azure.aks.cluster_autoscaler
1.0.16
guard
cloud.azure.aks.guard
1.0.16
Policy
cloud.azure.aks.policy
1.0.16
Administrative
cloud.azure.aks.administrative
1.0.16
Microsoft.Network
APPLICATIONGATEWAYS
ApplicationGatewayAccessLog
cloud.azure.appgateway.access_log
1.0.16
ApplicationGatewayFirewallLog
cloud.azure.appgateway.firewall_log
1.0.16
Policy
cloud.azure.appgateway.policy
1.0.16
Administrative
cloud.azure.appgateway.administrative
1.0.16
AZUREFIREWALLS
AzureFirewallApplicationRule
cloud.azure.firewall.application_rule
1.0.16 (overwrite previous)
AzureFirewallNetworkRule
cloud.azure.firewall.network_rule
1.0.16 (overwrite previous)
AzureFirewallDnsProxy
cloud.azure.firewall.dns_proxy
1.0.16 (overwrite previous)
Microsoft.Storage
STORAGEACCOUNTS
Administrative
cloud.azure.storage.administrative
1.0.16
ResourceHealth
cloud.azure.storage.resourcehealth
1.0.16
Microsoft.Web
SITES
Administrative
cloud.azure.appservice.calculated_category
1.0.16
Policy
cloud.azure.appservice.policy
1.0.16
Microsoft.ContainerRegistry
REGISTRIES
ContainerRegistryLoginEvents
cloud.azure.contregistry.login
1.0.16
Microsoft.DBforPostgreSQL
SERVERS
PostgreSQLLogs
cloud.azure.postgresql.events
1.0.16
Microsoft.Compute
VIRTUALMACHINES
Administrative
cloud.azure.vm.administrative
1.0.16
ResourceHealth
cloud.azure.vm.resourcehealth
1.0.16
Policy
cloud.azure.vm.policy
1.0.16
Recommendation
cloud.azure.vm.recommendation
1.0.16
VIRTUALMACHINESCALESETS
Administrative
cloud.azure.vmscalesets.administrative
1.0.16
ResourceHealth
cloud.azure.vmscalesets.resourcehealth
1.0.16
Policy
cloud.azure.vmscalesets.policy
1.0.16
Autoscale
cloud.azure.vmscalesets.autoscale
1.0.16
Microsoft.DataFactory
FACTORIES
Administrative
cloud.azure.datafactory.administrative
1.0.16
Microsoft.Insights
ACTIVITYLOGALERTS
Alert
cloud.azure.monitor.alert
1.0.16
Microsoft.Security
LOCATIONS
Security
cloud.azure.securitycenter.security
1.0.16
Microsoft.KeyVault
VAULTS
AuditEvent
cloud.azure.keyvault.audit
1.0.16
Administrative
cloud.azure.keyvault.administrative
1.0.16
Policy
cloud.azure.keyvault.policy
1.0.16
Microsoft.aadiam
<empty>
SignInLogs
cloud.azure.ad.signin
1.0.16
AuditLogs
cloud.azure.ad.audit
1.0.16
If none of the previous patterns are matched, the following ones will be applied:
Provider
Service
Category
Devo table
*
*
Administrative
cloud.azure.others.administrative
Autoscale
cloud.azure.others.autoscale
Policy
cloud.azure.others.policy
Recommendation
cloud.azure.others.recommendation
ResourceHealth
cloud.azure.others.resourcehealth
The basic type detection will be applied for other values:
Message type
Devo table
event
cloud.azure.other.events
metric
cloud.azure.eh.metrics
Version 1.0.15
Provider
Service
Category
Devo table
Since version
Microsoft.Network
AZUREFIREWALLS
AzureFirewallApplicationRule
cloud.azure.eh.firewall
1.0.15
AzureFirewallNetworkRule
cloud.azure.eh.firewall
1.0.15
AzureFirewallDnsProxy
cloud.azure.eh.firewall
1.0.15
The basic type detection will be applied for other values:
Message type
Devo table
event
cloud.azure.eh.events
metric
cloud.azure.eh.metrics
Version 1.0.0
It doesn’t contain a full message auto-categorization functionality, just the detection of the message type (metric or event):
Message type
Devo table
event
cloud.azure.eh.events
metric
cloud.azure.eh.metrics
Vendor setup
The Microsoft Azure collector centralizes the data with an Event Hub using the Azure SDK. To use it, you need to configure the resources in the Azure Portal and set the right permissions to access the information.
To log in to the Azure subscription, the collector uses a Service Principal object. You need to get the subscription ID, Active Directory ID, Application ID (service principal identification), and the client secret (service principal "password"). To get them, follow these steps:
Log in to your Azure account and search for Azure Active Directory.
Now, click App registrations in the left menu and click the app (or Service Principal) that you are going to use.
In the Overview area, find the Application (client) ID and the Directory (tenant) ID.
Now click Certificates & Secrets on the menu and create a new client secret by clicking the New client secret button. Don't forget to save the client secret value, it will be only shown upon creation.
Get the subscription ID by searching for Subscriptions on the home page.
Find the correct subscription and note down the subscription ID.
Setting up permissions
You need Administration permissions to follow these steps.
After creating the App registration (or Service Principal), go to the desired Resource Group (or subscription if you want to retrieve metrics from all the available virtual machines).
Select Access control (IAM) in the left menu and click Add.
Select at least the Reader role and choose the previously created App registration.
Confirm the changes.
Event Hub events
Getting credentials (Storage Account) (Optional)
If you want to use Azure Blob Storage for checkpointing purposes, you need to create a storage account to store the checkpoints. If you do not wish to use Azure Blob storage (i.e. you will use Devo local persistence), you can skip the Blob Storage configuration steps.
Connection string
From the left portal menu, select Storage accounts to display a list of your storage accounts. If the portal menu isn't visible, select the menu button to toggle it on.
On the Storage accounts page, select Create.
After the storage account is created, select it from the list of storage accounts, click on Access keys in the left menu, and copy the connection string.
Role assignment
Alternatively, users can grant the necessary permissions to the registered application to access the Event Hub without using the RootManageSharedAccessKey. Roles can be assigned in a variety of ways (e.g. inherited from the subscription group), but the following steps will show how to assign the necessary roles directly to the Storage Account.
Repeat steps 1-2 from the Connection String section to create the Storage Account.
In the Storage Account, click Access control (IAM) in the left menu, click + Add, and click Add Access Role Assignment.
Search for either the Storage Blob Data Contributor or Storage Blob Data Owner role and select it and then click Next.
Click + Select members and search for the previously created App registration, select it, click Next.
Click Review + Assign.
Getting credentials (Event Hubs)
Users can either obtain a connection string or use Role Assignments to allow the collector to access the Event Hub.
Connection string
In your Azure account, search for the Event Hubs service and click on it.
Create an Event Hub resource per region (repeat the steps below for each region):
Click Add.
Fill the mandatory fields keeping in mind that the Event Hub must be in the same region as the resources that you are going to monitor (and only need one per region). The Throughput Units option refers to the ingress/egress limit in MB/s (each unit is 1 MB/s or 1000 events/second ingress, 2 MB/s, or 4096 events/second egress). You should adjust it according to the data volume (this can be modified later).
The previous steps create an EventHub namespace; now go to Event Hubs, search the created one and click on it.
Now click on the + Event Hub button and create a new resource. You only need to fill the Name and Partition Count fields (the Partition Count field will divide the data into different partitions to make it easier to read large volumes of data). Write down the EventHub name to be used later in the configuration file.
Once the Event Hub is created in the namespace, click it and select Consumer Group in the left menu. Note that a dedicated Consumer Group for Devo needs to be created if the existing consumer groups are already in use.
Here you will see the Event Hub consumer groups. This will be used by the collector (or other applications) for reading data from the Event Hub. Write down the Consumer group name that you will use later in the configuration file. Now, in the Event Hub Namespace, click on Shared access policies, search the default policy named RootManageSharedAccessKey and click it.
Copy and write down the primary (or secondary) connection string to be used later in the configuration file.
Role assignment
Alternatively, users can grant the necessary permissions to the registered application to access the Event Hub without using the RootManageSharedAccessKey. Roles can be assigned in a variety of ways (e.g. inherited from the subscription group), but the following steps will show how to assign the necessary roles directly to the Event Hub Namespace.
Repeat steps 1-2.7 from the previous section to create the Event Hub.
In the Event Hub Namespace, click Access control (IAM) in the left menu, click + Add, and click Add Access Role Assignment.
Search for either the Azure Event Hubs Data Receiver or Azure Event Hubs Data Owner role and select it and then click Next.
Click + Select members and search for the previously created App registration, select it, click Next.
Click Review + Assign.
Setting up the Event Hubs
Now, search the Monitor service and click on it.
Click the Diagnostic Settings option in the left area.
A list of the deployed resources will be shown. Search for the resources that you want to monitor, select them, and click Add diagnostic setting.
Type a name for the rule and check the required category details (logs will be sent to the cloud.azure.eh.events table, and metrics will be sent to the cloud.azure.eh.metrics table).
Check Stream to an Event Hub, and select the corresponding Event hub namespace, Event hub name, and Event hub policy name.
Click Save to finish the process.
Event Hub Auto Discover
To configure access to event hubs for the auto-discovery feature, you need to grant the necessary permissions to the registered application to access the Event Hub without using the RootManageSharedAccessKey. Furthermore, the auto-discovery feature will enumerate a namespace and resource group for all available event hubs and optionally create consumer groups (if the configuration specifies a consumer group other than $Default and that consumer group does not exist when he collector connects to the event hub) and optionally create Azure Blob Storage containers for checkpointing purposes (if the user specifies a storage account and container in the configuration file).
Role assignment (Namespace)
Repeat the steps from the Event Hubs Role Assignment section, except that the necessary role is the Azure Event Hubs Namespace Data Owner role. This allows the collector to enumerate the event hubs in the namespace and create consumer groups if necessary.
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting
Details
tenant_id
The Azure application tenant ID.
client_id
The Azure application client ID.
client_secret
The Azure application client secret.
subscription_id
The Azure application subscription ID.
For Azure Event Hub, it is enough with the event hub name and the connection string (and optionally consumer group). No credentials are required.
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
VM Metrics (vm_metrics)
Internal process and deduplication method
All VM metrics data are pulled with a time grain value of PT1M (1 minute). The collector polls for all available VM resource IDs and then pulls the metrics for each resource ID. Checkpoints are persisted to ensure that duplicate data is not sent to Devo.
Devo categorization and destination
All events of this service are ingested into the table cloud.azure.vm.metrics_simple
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Event Hubs (event_hubs)
General principles
Understanding the following principles of Azure Event Hubs is crucial:
Consumer Groups: A single event hub can have multiple consumer groups, each representing a separate view of the event stream.
Checkpointing: The SDK supports checkpoint mechanisms to balance the load among consumers for the same event hub and consumer group. Supported mechanisms include:
Azure Blob Storage Checkpoint: Recommended to use one container per consumer group per event hub.
Partition Restrictions: Azure Event Hubs limits the number of partitions based on the event hub tier. For quotas and limits, refer to the official documentation.
Configuration options
Devo supports various configurations to cater to different Azure setups.
Event Hubs authentication configuration
Event Hubs authentication can be via connection strings or client credentials (assigning the Azure Event Hubs Data Receiver role). Preference is given to connection string configuration when both are available.
Event Hub Details: Retrieve details, including partition count.
Client Creation: For each queue, create Event Hub consumer clients.
If the user configured a client_thread_limit, clients will be created for each event hub partition up to the specified limit. In these cases, the consumer clients will not be explicitly assigned partitions and load balancing and partition assignment will be performed dynamically by the event hub SDK utilizing the checkpoints.
If the user did not configure a client_thread_limit, the collector will create a consumer client for each partition and explicitly assign the respective partition ID to the consumer client.
Event Fetching: Enable consumers to start fetching events.
Load balancing and event processing occurs throughout the fetching loop.
Event Processing: Events are fetched in batches. Records are extracted from event batches, deduplicated, tagged, and sent to Devo.
Checkpointing: After processing an event batch, checkpoints are updated so that the events will not be fetched again.
Configuration considerations
Multi-pod mode
While multi-pod mode is supported and represents the highest throughput possible for the collector, it requires the user to configure the collector in a specific manner to ensure that the collector operates efficiently and does not send duplicate events to Devo (see below). In most cases, multi-pod mode is unnecessary.
High Throughput: Multi-pod mode allows potentially the highest throughput.
Multi-pod mode is recommended for scenarios in which the user has more partitions than can be supported on a single collector instance.
Consumer Client Thread Limit: The user should specify a client_thread_limit to ensure that the collector utilizes load balancing instead of explicitly assigning partition IDs to the consumer clients.
In load-balancing mode, having fewer consumer clients than partitions is allowable, but not as efficient as some consumer clients will fetch events from multiple partitions.
In load-balancing mode, having more consumer clients than partitions is allowable, but not as efficient as some consumer clients will not be assigned any partitions.
The most efficient design is to ensure that there are as many consumer clients as there are partitions distributed amongst the pods. The easiest way to achieve this is to set the client_thread_limit to 1 and creating as many pods as there are partitions.
Azure Blob Storage Checkpointing: Required for multi-pod mode.
Warning: Running in multi-pod with local checkpointing will result in duplicate events being sent to Devo because the load balancing operation will have no visibility of the other pods' checkpoints.
Standard mode
Both checkpointing options are supported. In standard mode, the collector will automatically create one consumer client thread per partition per event hub.
If the event hubs you wish to fetch data from have too many partitions that can be supported on a single instance (i.e. you have 100 event hubs each with 32 partitions, therefore the collector attempts to create 3200 consumer clients), then you should create multiple collector instances and configure each one to fetch from a subset of the desired events hubs.
Internal process and deduplication method
The collector uses the event_hubs service to pull events from the Azure Event Hubs. Each queue in the event_hubs service represents an event hub that is polled for events.
Collector deduplication mechanisms
Events are deduplicated using the duplicated_messages_mechanism parameter. There are two methods available:
Local Deduplication: Ensures that subsequent duplicate events from the same event hub are not sent to Devo. This method operates individually within each consumer client.
Global Deduplication: Utilizes a shared cache across all event hub consumers for a given collector. As events are ingested into Devo, the collector checks if the event has already been consumed by another event hub consumer. The event will not be sent to Devo if it has already been consumed. The global cache tracks the last 1000 events for each consumer client.
If the global deduplication method is selected, the collector will automatically employ the local deduplication method as well.
Checkpointing mechanisms
The collector offers two distinct methods for checkpointing, each designed to prevent the re-fetching of events from Azure Event Hubs. These mechanisms ensure efficient event processing by maintaining a record of the last processed event in each partition.
Local Persistence Checkpointing
Overview: By default, the collector employs local persistence checkpointing. This method is designed to keep track of the last event offset within each partition of an Event Hub, ensuring events are processed once without duplication.
How It Works: As the collector consumes messages from an Event Hub, it records the offset of the last processed event locally. On subsequent pulls from the Event Hub, the collector resumes processing from the next event after the last recorded offset, effectively skipping previously processed events.
Use Case: Ideal for single-instance deployments where all partitions of an Event Hub are managed by a single collector instance.
Azure Blob Storage Checkpointing
Overview: As an alternative to local persistence, the collector can be configured to use Azure Blob Storage for checkpointing. This approach leverages Azure's cloud storage to maintain event processing state.
Configuration:
Option 1: Specify both an Azure Blob Storage account and container name. This method requires the collector to have appropriate access permissions to the specified Blob Storage account.
Option 2: Provide an Azure Blob Storage connection string and container name. This method is straightforward and recommended if you have the connection string readily available.
Benefits:
Multi-pod Support: Enables the collector to operate in a distributed environment, such as Kubernetes, where multiple instances (pods) of the collector can run concurrently. Checkpointing data stored in Azure Blob Storage ensures that each instance has access to the current state of event processing, facilitating efficient load balancing and event partition management.
Durability: Utilizes Azure Blob Storage's durability and availability features to safeguard checkpointing data against data loss or corruption.
Use Case: Recommended for environments requiring multi-pod deployment or when a user prefers to centralize checkpointing within their Azure infrastructure.
Event Hubs Auto Discover (event_hubs_auto_discover)
General principles
Refer to Event Hubs - General Principles for general principles.
Configuration options
Devo supports only one for this service. Connection strings are not supported.
Event Hubs Auto Discover authentication configuration
Event Hubs authentication can be via connection strings or client credentials (assigning the Azure Event Hubs Data Receiver role).
Preference is given to connection string configuration when both are available.
The collector uses the event_hubs_auto_discover to dynamically query a given resource group and namespace for all available event hubs.
All deduplication methods and checkpointing methods listed in the event_hubs service apply; however, there are some additional considerations one should make when configuring the event_hubs_auto_discover service.
The event_hubs_auto_discover service will effectively restart all event hub consumers after one hour (this time can be overridden via the override_consumer_client_ttl_seconds_value parameter.) On restart, the collector will re-discover all available event hubs and begin pulling data again. Any event hubs that might have been created between the last run and the current run will be discovered and pulled from.
Due to the nature of this service, if a user has configure Azure Blob Storage checkpointing, the collector will attempt to create containers in the configured Azure Blob storage account. If the configured credentials do not have write access to the storage account, an error will be presented to the logs and indicate that the user must grant write access to the credentials.
Checkpointing
The collector supports two forms of checkpointing.
Local persistence checkpointing
By default, the collector will utilize local persistence checkpointing to ensure that events are not fetched multiple times from a given partition in a given event hub. The collector will store the last event offset as messages are consumed.
Azure Blob Storage checkpointing
Optionally, users can specify an Azure Blob Storage account or an Azure Blob Storage connection string to use Azure Blob Storage checkpointing. This allows the collector to run in multi-pod mode and all checkpointing data is stored within the Azure Storage account.
Unlike the event_hubs service, the event_hubs_auto_discover service will create containers for the discovered event hubs in the configured Azure Blob Storage account. The containers are prefixed with devo- (though this value can be overridden in the configuration) and a hash calculated from the resource group, namespace, event hub name, and consumer group. This hash is used to ensure that the container name is unique and does not conflict with other container names and is within the character limit for Azure container names.
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Common logic
Error type
Error ID
Error message
Cause
Solution
InitVariablesError
1
Invalid start_time_in_utc: {ini_start_str}. Must be in parseable datetime format.
The configured start_time_in_utc parameter is a non-parseable format.
Update the start_time_in_utc value to have the recommended format as indicated in the guide.
InitVariablesError
2
Invalid start_time_in_utc: {ini_start_str}. Must be in the past.
The configured start_time_in_utc parameter is a future date.
Update the start_time_in_utc value to a past datetime.
PullError
350
Could not match tag to record and no default tag provided: {record}
Advanced tagging configured but no default tag provided and record did not match any of tag pathways
Provide default tag in advanced tag mapping object
ApiError
401
An error occurred while trying to authenticate with the Azure API. Exception: {e}
The collector is unable to authenticate with the Azure API.
Check the credentials and ensure that the collector has the necessary permissions to access the Azure API.
ApiError
410
An error occurred while trying to check if container '{container_name}' exists. Ensure that the blob storage account name or connection string is correct. Exception: {e}
The collector was unable to locate the specified blob storage container name.
Ensure the container exists and the credentials have READ access to the container
ApiError
411
An error occurred while trying to check if container '{container_name}' exists. Ensure that the application has necessary permissions to access the containers. Exception: {e}
The collector was unable to access the specified blob storage container name.
Ensure the container exists and the credentials have READ access to the container
ApiError
412
An error occurred while trying to create container '{container_name}'. Ensure that the application has necessary permissions to create containers. Exception: {e}
The collector was unable to create the container for the auto discover service and the user indicated to use Azure Blob Storage checkpointing.
Ensure the credentials have WRITE access to the container storage account.
ApiError
420
An error occurred while trying to get consumer group '{consumer_group_name}'. Exception: {e}
The collector was unable to access the specified consumer group name.
Ensure the consumer group exists and the credentials have READ access to the consumer group
ApiError
421
An error occurred while trying to create consumer group '{consumer_group_name}'. Ensure that the application has necessary permissions to create consumer groups. Exception: {e}
The collector was unable to create the consumer group for the auto discover service.
Ensure the credentials have WRITE access to the event hub namespace or use the $Default consumer group.
Typical issues
CBS token error - This issue happens usually when the connection string includes the event hub namespace name instead of the event hub name. Both values are usually different and it is easy to mix up both. You can find a explanation here.
Delayed events - You can use the @devo_event_enqueued_time value in the table cloud.azure to check the time that the events are queued in Azure. The delayed events can be caused by Event Hub itself (high enqueued time), or by lack of processing capacity of collector. In this case, it is necessary to add more collector instances, or to create a collector for each partition.
Duplicated events - Adjust the value of the config parameter duplicated_messages_mechanism_value according to your deployment. If you are running several instances, change the value to local. See [Internal Process and Deduplication Method](Internal Process and Deduplication Method) for more details.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Verify collector operations
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-test-local.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> "\etc\devo\job" does not exists
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> "\etc\devo\collector" does not exists
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "C:\git\collectors2\devo-collector-<name>\config\config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
2023-01-10T15:22:57.171 WARNING MainProcess::MainThread -> [WARNING] Illegal global setting has been ignored -> multiprocessing: FalseEvents delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:
Sender services
Description
internal_senders
In charge of delivering internal metrics to Devo such as logging traces or metrics.
standard_senders
In charge of delivering pulled events to Devo.
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace
Description
Number of available senders: 1
Displays the number of concurrent senders available for the given Sender Service.
sender manager internal queue size: 0
Displays the items available in the internal sender queue.
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.
Total number of messages sent: 44, messages sent since "2022-06- 28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)
Displays the number of events from the last time and following the given example, the following conclusions can be obtained:
44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 0 events required 0.007 seconds to be delivered.
By default these traces will be shown every 10 minutes.
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
This section will walk you through the process of updating your configuration from the old version (1.x.x) to the new version (2.0.0). The new version introduces significant improvements and changes to the configuration style to enhance performance, usability, and security.
Overview of changes
The new configuration format introduces several key changes:
Multiple inputs: The configuration now supports multiple inputs to better represent the different data sources and access mechanisms (azure and azure_event_hub)
Rename credential config parameters: The credentials configuration field names now follow names that are consistent with Microsoft Azure documentation: tenant_id, client_id, client_secret.
Azure Blob storage checkpoint support: The configuration now accepts Azure Blob Storage related parameters in the queue-specific configuration: blob_storage_connection_string, blob_storage_container_name, blob_storage_account_name.
Moved VM metrics to dedicated service: The VM metrics input has been moved to a dedicated service. The customer service configuration is no longer valid.
Moved Event Hub to dedicated service: The Event Hub input has been moved to a dedicated service. The customer service configuration is no longer valid.
Preparing for migration
Before starting the migration process, we recommend the following steps:
Backup your current configuration: Always ensure you have a backup of your existing configuration files to prevent any data loss.
Review the new configuration documentation: Familiarize yourself with the new configuration options available in version 2.0.0.
Migration steps
Step 1: Update credential configuration parameter field names
The credential configuration field names have been updated:
active_directory_id → tenant_id
secret → client_secret
app_id → client_id
An example of the old and new configuration is shown below:
# Old Version (1.x.x)
credentials:
app_id: <app_id_value>
active_directory_id: <active_directory_id_value>
subscription_id: <subscription_id_value>
secret: <secret_value>
user_guide.md 2024-05-21
57 / 60
↓
# New Version (2.0.0)
credentials:
client_id: <client_id_value>
tenant_id: <tenant_id_value>
subscription_id: <subscription_id_value>
client_secret: <client_secret_value>
Step 2: Update VM metrics configuration
The VM Metrics service has been moved to the azure input and a dedicated vm_metrics service.
An example of the new configuration is shown below:
If you wish to continue from the old configuration, you must input the time of the latest in Devo in the start_time_in_utc field to indicate the time from which the puller will start collecting data.
Step 3: Update Event Hub Configuration
The Event Hub service(s) have been moved to the azure_event_hub input and a dedicated event_hub service.
The new configuration now accepts the blob_storage_connection_string, blob_storage_container_name, and blob_storage_account_name parameters in the queue-specific configuration. These parameters are new, optional, and only required for those users who wish to leverage the Azure Blob Storage for checkpoint. This guide focuses on migrating the configuration from the old version to the new version -- for this reason, the new Azure Blob Storage checkpoint parameters are not relevant to older configurations because they use local, file-based checkpointing.
By default, the collector will begin pulling from the latest event in the queue if there is not already a pre-existing checkpoint. To ensure your migrated collectors fetch from the last event previously sent to Devo, identify the date time of the last event in Devo for the relevant queue and input it into the override_starting_position field in the format %Y-%m-%dT%H:%M:%SZ. When the collector begins pulling from the queue, the collector will begin fetching from the indicated date time for the first checkpoint.
Step 4: Example before and after configuration
Putting it all together, see below for an example of the old and new configuration:
# New Version (2.0.0)
inputs:
azure:
id: 100001
enabled: true
credentials:
subscription_id: subscription_id_acme
client_id: app_id_acme
client_secret: secret_acme
tenant_id: active_directory_id_acme
environment: test-env
services:
vm_metrics:
request_period_in_seconds: 300
azure_event_hub:
id: 100001
enabled: true
credentials:
subscription_id: subscription_id_acme
client_id: app_id_acme
client_secret: secret_acme
tenant_id: active_directory_id_acme
environment: test-env
services:
event_hubs:
queues:
queue_a:
event_hub_name: the-event-hub-name
event_hub_connection_string: the-connection-string
consumer_group: the-consumer-group
events_use_autocategory: true
compatibility_version: 1.2.0
duplicated_messages_mechanism: global
override_starting_position: "2022-01-01T00:00:00Z" # Replace with the datetime of the last event in Devo.
Otherwise, collector pulls from latest event for the first checkpoint.
Tag mapping configuration guide
The events from Event Hubs are by default auto-categorized to Devo tags according the values explained here. But sometimes it can be needed to change this categorization or create a new categorization for a new kind of events. It is possible to change the categories without creating a new version of the collector, editing the config file.
This guide explains how to configure mapping using the tag parameter in the YAML configuration.
Overview
By default, the override_tag parameter accepts a simple string that will be applied to all records; however, the advanced override_tag parameter allows you to define a default tag and a set of tag mapping rules based on JMESPath expressions. The collector will use these rules to assign tags to records based on their content.
You can find a tutorial and a complete reference for JMESPath here.
Use the default_tag parameter to specify the default tag that will be applied to records that do not match any JMESPath expression.
JMESPath references (Optional)
Define reusable JMESPath expressions in the jmespath_refs section.
These expressions can be referenced in the tag_map section using placeholders (e.g., {events_base}).
Tag map
Define a list of tag mapping rules in the tag_map section.
Each rule consists of a jmespath expression and a corresponding tag.
The jmespath expression is evaluated against each record, and if it matches, the corresponding tag is applied to the record.
The tag value can include placeholders (e.g., {queue_name}, {collector_version}) that will be substituted with values from the record itself or the collector variables.
Example simple configuration (used internally by the Google Workspace Logs in BigQuery collector)
override_tag:
default_tag: my.app.gsuite_activity.{record_type}
tag_map:
- jmespath: "[?record_type == 'gmail']"
tag: cloud.gcp.bigquery.gmailExample advanced configuration (used internally by the Azure collector)
Example advanced configuration (used internally by the Azure collector)
The collector evaluates each record against the JMESPath expressions in the tag_map section, in top-down order.
If a record matches a JMESPath expression, the corresponding tag is applied, and the record is not evaluated against subsequent expressions.
If a record does not match any JMESPath expression, the default_tag is applied.
Sending records
After all evaluations are made for a given recordset, the collector groups the records by their assigned tags.
The collector sends each group of records to Devo on a per-tag basis.
By configuring the override_tag parameter according to this guide, you can effectively map tags to records based on their content using JMESPath expressions.
Change log
Release
Released on
Release type
Details
Recommendations
v2.0.0
IMPROVEMENTS
Improvements
Complete reimplementation of the collector, refactoring all the services
Recommended version
v1.9.0
IMPROVEMENTS
Improvements
Updated DCSDK from 1.10.3 to 1.11.0
Resolution the UTF16 issues.
Fixed some bug related to the development.
Update
v1.8.0
IMPROVEMENTS
BUG FIXING
Improvements
Update DCSDK from 1.9.2 to 1.10.3:
Updated DevoSDK to v5.1.9
Fixed some bug related to development on MacOS
Added an extra validation and fix when the DCSDK receives a wrong timestamp format
Added an optional config property for use the Syslog timestamp format in a strict way
Bug fixing
A bug related to UTF-16 causing the collector to stop sending events
Update
v1.7.1
BUG FIXING
Bug fixing
Azure metrics were using the incorrect timestamp format which caused logs to go to unknown
Update
v1.7.0
IMPROVEMENTS
BUG FIXING
Improvements
Update DCSDK from 1.8.0 to 1.9.2:
Upgrade internal dependencies
Store lookup instances into DevoSender to avoid creation of new instances for the same lookup
Ensure service_config is a dict into templates
Ensure special characters are properly sent to the platform
Changed log level to some messages from info to debug
Changed some wrong log messages
Upgraded some internal dependencies
Changed queue passed to setup instance constructor
Update internal Azure libraries
Bug fixing
Enhancement for event category calculation
Update
v1.6.0
IMPROVEMENTS
BUG FIXING
Improvements
Update DCSDK from 1.3.0 to 1.8.0:
Added log traces for knowing the execution environment status (debug mode)
Fixes in the current puller template version
The Docker container exits with the proper error code
New controlled stopping condition when any input thread fatally fails
Improved log trace details when runtime exceptions happen
Refactored source code structure
New "templates" functionality
Functionality for detecting some system signals for starting the controlled stopping
Input objects sends again the internal messages to devo.collectors.out table
Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo
Refactored source code structure
Changed way of executing the controlled stopping
Minimized probabilities of suffering a DevoSDK bug related to "sender" to be null
Ability to validate collector setup and exit without pulling any data
Ability to store in the persistence the messages that couldn't be sent after the collector stopped
Ability to send messages from the persistence when the collector starts and before the puller begins working
Ensure special characters are properly sent to the platform
Added a lock to enhance sender object
Added new class attrs to the setstate and getstate queue methods
Fix sending attribute value to the setstate and getstate queue methods
Added log traces when queues are full and have to wait
Added log traces of queues time waiting every minute in debug mode
Added method to calculate queue size in bytes
Block incoming events in queues when there are no space left
Send telemetry events to Devo platform
Changed
Upgraded internal Python dependency Redis to v4.5.4
Upgraded internal Python dependency DevoSDK to v5.1.3
Fixed obfuscation not working when messages are sent from templates
Bug fixing
Updated Azure libraries for Python are updated to share common cloud patterns.
Change in the authentication mechanism:
Previous version: Used ServicePrincipalCredentials in azure.common to authenticate to Azure.
New version: Uses the azure.identity library to provide unified authentication for all Azure SDKs.
Update
v1.5.0
BUG FIXING
Bug fixing
Accept a batch of events that come as an array.
Filter out non-VM-related events in the SourceSystem branch.
Update
v1.4.1
IMPROVEMENTS
Improvements
Upgraded underlay IFC SDK v1.3.0 to v1.4.0.
Updated the underlying DevoSDK package to v3.6.4 and dependencies, this upgrade increases the resilience of the collector when the connection with Devo or the Syslog server is lost. The collector is able to reconnect in some scenarios without running the self-kill feature.
Support for stopping the collector when a GRACEFULL_SHUTDOWN system signal is received.
Re-enabled the logging to devo.collector.out for Input threads.
Improved self-kill functionality behavior.
Added more details in log traces.
Added log traces for knowing system memory usage.
Update
v1.4.0
IMPROVEMENTS
Improvements
New events types are accepted for the service vm_events autocategorizer.
cloud.azure.vm.securityevent:
Type: Event
EventID: all
EventLog: Security
cloud.azure.vm.applicationevent:
Type: Event
EventID: all
EventLog: Application
cloud.azure.vm.systemevent:
Type: Event
EventID: all
EventLog: System
Update
v1.3.2
BUG FIXING
Bug fixing
A configuration bug has been fixed to enable the autocategorization of the following events
Now, in the Event Hub Namespace, click on Shared access policies, search the default policy named RootManageSharedAccessKey and click it.
