Azure collector parameters

Example:

{
 "global_overrides": {
 "debug": false
 },
 "inputs": {
 "azure": {
 "id": "<short_unique_id>",
 "enabled": true,
 "credentials": {
 "subscription_id": "<subscription_id_value>",
 "client_id": "<client_id_value>",
 "client_secret": "<client_secret_value>",
 "tenant_id": "<tenant_id_value>"
 },
 "environment": "<environment_value>",
 "services": {
 "vm_metrics": {
 "request_period_in_seconds": "<request_period_in_seconds_value>",
 "start_time_in_utc": "<start_time_in_utc_value>",
 "include_resource_id_patterns": [
 "<include_resource_id_patterns_values>"
 ],
 "exclude_resource_id_patterns": [
 "<exclude_resource_id_patterns_values>"
 ]
 }
 }
 },
 "azure_event_hub": {
 "id": "<short_unique_id>",
 "enabled": true,
 "credentials": {
 "subscription_id": "<subscription_id_value>",
 "client_id": "<client_id_value>",
 "client_secret": "<client_secret_value>",
 "tenant_id": "<tenant_id_value>"
 },
 "environment": "<environment_value>",
 "services": {
 "event_hubs": {
 "override_pull_report_frequency_seconds": "<override_pull_report_frequency_seconds_value>",
 "override_consumer_client_ttl_seconds": "<override_consumer_client_ttl_seconds_value>",
 "queues": {
 "<queue_name_value>": {
 "namespace": "<namespace_value>",
 "event_hub_name": "<event_hub_name_value>",
 "event_hub_connection_string": "<event_hub_connection_string_value>",
 "consumer_group": "<consumer_group_value>",
 "blob_storage_connection_string": "<blob_storage_connection_string_value>",
 "blob_storage_container_name": "<blob_storage_container_name_value>",
 "blob_storage_account_name": "<blob_storage_account_name_value>",
 "compatibility_version": "<compatibility_version_value>",
 "duplicated_messages_mechanism": "<duplicated_messages_mechanism_value>",
 "override_starting_position": "<override_starting_position_value>",
 "override_tag": "<override_tag_value>",
 "extend_tag": "<extend_tag_value>",
 "client_thread_limit": "<client_thread_limit_value>",
 "uamqp_transport": "<uamqp_transport_value>",
 "partition_ids": ["<partition_id>"]
 }
 }
 },
 "event_hubs_auto_discover": {
 "resource_group": "<resource_group_value>",
 "namespace": "<namespace_value>",
 "blob_storage_account_name": "<blob_storage_account_name_value>",
 "blob_storage_connection_string": "<blob_storage_connection_string_value>",
 "consumer_group": "<consumer_group_value>",
 "duplicated_messages_mechanism": "<duplicated_messages_mechanism_value>",
 "override_pull_report_frequency_seconds": "<override_pull_report_frequency_seconds_value>",
 "override_consumer_client_ttl_seconds": "<override_consumer_client_ttl_seconds_value>",
 "override_starting_position": "<override_starting_position_value>",
 "override_blob_storage_container_prefix": "<override_blob_storage_container_prefix_value>",
 "client_thread_limit": "<client_thread_limit_value>",
 "uamqp_transport": "<uamqp_transport_value>"
 }
 }
 }
 }
}

Parameter	Data type	Requirement	Value range / Format	Description
`short_unique_id`	`str`	Mandatory	Min length: 1, Max length: 5	Short, unique ID for input service, used in persistence addressing. Avoid duplicates to prevent collisions.
`tenant_id_value`	`str`	Mandatory	Min length: 1	Tenant ID for Azure authentication.
`client_id_value`	`str`	Mandatory	Min length: 1	Client ID for Azure authentication.
`client_secret_value`	`str`	Mandatory	Min length: 1	Client secret for Azure authentication.
`subscription_id_value`	`str`	Mandatory	Min length: 1	Azure subscription ID.
`environment_value`	`str`	Optional	Min length: 1	Differentiates environments (e.g., dev, prod). Remove if unused.
`request_period_in_seconds_value`	`int`	Optional	Min: 60	Custom period in seconds between data pulls, overriding default (300s).
`start_time_in_utc_value`	`str`	Optional	UTC datetime format: `%Y-%m-%dT%H-%M-%SZ`	Custom start date for data retrieval, for historical data download. Remove if unused.
`include_resource_id_patterns_values`	`[str]`	Optional	Glob patterns e.g., `["VM-GROUP-1"]`	Includes resources matching patterns. Remove if unused.
`exclude_resource_id_patterns_values`	`[str]`	Optional	Glob patterns e.g., `["VM-GROUP-1"]`	Excludes resources matching patterns. Remove if unused.
`queue_name_value`	`str`	Mandatory	Min length: 1	Name for the queue, appears in related logs.
`event_hub_name_value`	`str`	Mandatory	Min length: 1	Name of the Event Hub to pull events from.
`event_hub_connection_string_value`	`str`	Mandatory	Min length: 1	Connection string for the Event Hub.
`consumer_group_value`	`str`	Optional	Min length: 1, Default: `$Default`	Consumer group for the Event Hub. Defaults to `$Default`.
`events_use_autocategory_value`	`bool`	Optional	Default: `true`	Enables auto-tagging of events. This value is always true.
`blob_storage_connection_string_value`	`str`	Optional	Min length: 1	Connection string for blob storage, optional for Azure Blob Storage checkpointing.
`blob_storage_container_name_value`	`str`	Optional	Min length: 1	Blob storage container name, required if using Azure Blob Storage checkpointing.
`blob_storage_account_name_value`	`str`	Optional	Min length: 1	Blob storage account name, alternative to using connection string for checkpointing.
`compatibility_version_value`	`str`	Optional	Version strings	Compatibility version for event processing.
`duplicated_messages_mechanism_value`	`str`	Optional	One of: `"local"`, `"global"`, `"none"`	Deduplication mechanism for messages: local, global, or none.
`override_starting_position_value`	`str`	Optional	One of: `"-1"`, `"@latest"`, `"[UTC datetime value]"`	Starting position for event fetching: from the beginning of available data (-1), from the latest data fetched (@fetched), or a specific datetime (%Y-%m- %dT%H-%M-%SZ format).
`override_tag_value`	`str`	Optional	Tag-friendly string	Optional tag to override the default tagging mechanism. See Event Hubs Tagging Configuration.
`extend_tag_value`	`str`	Optional	Object that can include any of the following properties: default_tag, tag_map, jmespath_refs	Advanced feature. Allows users to add/update various properties of the tag. If the user utilized `override_tag` and configured a simple tag string, this parameter will have no effect. If supplied, `default_tag` overrides the default tag, `jmespath_refs` adds/updates jmespath substitution values, and `tag_map` will add/update various tag paths to the pre-existing tag map. See Event Hubs Tagging Configuration.
`override_pull_report_frequency_seconds_value`	`int`	Optional	Default: 60	Frequency in seconds for reporting pull statistics in logs.
`override_consumer_client_ttl_seconds_value`	`int`	Optional	Default varies by service	Time-to-live in seconds for consumer clients, after which the collector restarts the pull cycle.
`resource_group_value`	`str`	Mandatory	Min length: 1	Azure resource group for event hub discovery.
`namespace_value`	`str`	Mandatory	Min length: 1	Namespace within Azure for event hub discovery.
`override_blob_storage_container_prefix_value`	`str`	Optional	Min length: 3, Max length: 10; Default: `devo-`	Prefix for blob storage containers created by auto-discovery service. Remove if unused.
`uamqp_transport_value`	`bool`	Optional	Default: `false`	Allows users to override/force event hub SDK to use legacy UAMQP transport mechanism (true)instead of the default/current PyAMQP mechanism (false).
`<partition_ids>`	`str`	Optional	List of partition number, as `["1","3","5","7"]`	Allows to define which partitions are going to be connected by this instance of the collector. It overrides `client_thread_limit_value`
`client_thread_limit_value`	`int`	Optional	Min value: 1	Adv feature - most users should use `partition_ids` instead to explicitly define what partitions the collector instance will query. Number of consumer threads that the collector will create. By default, collector will create as many threads as there are consumers in the event hub.

Parameters marked as "Mandatory" are required for the collector's configuration. Optional parameters can be omitted or removed if not used, but they provide additional customization and control over the collector's behavior.

Event Hubs Tagging Configuration

Event Hubs supports multiple tagging parameters and formats to categorize and manage event data efficiently. Below are the configuration options for overriding, auto-categorizing, and extending tags.

The default configuration of the tag mapping can be found in this article.

Override tag

Advanced setting. Please consult to Devo support before use advanced tag map.

To customize the default tag behavior, users can configure the override_tag parameter within the Event Hub queue configuration. This parameter allows either a simple tag string or a more advanced tag mapping structure to be applied to all records.

The advanced tag map structure follows this format:

default_tag: A fallback tag applied to all records not matched by any tag_map entry.
tag_map: A list of tag entries, each containing a tag value and a JMESPath expression to match specific records.
jmespath_refs: Reference variables that can be used within JMESPath expressions in the tag_map. These act as reusable values within the tag map's matching logic.

override_tag:
 default_tag: "tag_value"
 tag_map:
 - tag: "tag_value"
 jmespath: "[?condition]"
 - tag: "tag_value"
 jmespath: "[?condition]"
 ...
 jmespath_refs:
 jmespath_ref_1: "{jmespath_expression_1}"
 jmespath_ref_2: "{jmespath_expression_2}"
 ...

"override_tag": {
 "default_tag": "tag_value",
 "tag_map": [
 {
 "tag": "tag_value",
 "jmespath": "[?condition]"
 },
 {
 "tag": "tag_value",
 "jmespath": "[?condition]"
 }
 .......
 ],
 "jmespath_refs": {
 "jmespath_ref_1": "{jmespath_expression_1}",
 "jmespath_ref_2": "{jmespath_expression_2}"
 }
 ........
 }

Auto-Category Tagging

From version 2.4 onwards, Auto Category is always enabled

Auto-category automatically appends pre-defined tags to the default tag (or the override_tag, if specified), enabling Azure events to be mapped dynamically to the appropriate Devo tag.

The system attempt to extract both the resource ID and the event category from the Azure event. If an event does not match any preconfigured tag mappings, it will be categorized under the following format: cloud.azure.{resource_id}.{category}.{queue_name}.

Auto-category tags are evaluated before the default or override tags.

Extend tag

Users can further customize tags by using the extend_tag parameter in the Event Hub queue configuration. This feature allows for the extension or updating of various tag properties. If override_tag is being used, the extend_tag will modify it; otherwise, it will extend the default tag.

The extend_tag parameter offers the following options:

default_tag: Replaces the existing default tag.
jmespath_refs: Adds or updates JMESPath substitution values.
tag_map: Adds or updates entries in the existing tag map. If an extend_tag entry matches an existing tag or JMESPath expression, that entry is replaced; otherwise, the new entry is appended.

Here is an example of extend_tag configuration:

Please note that the actual internal tag structure is not displayed in this guide as it is subject to change.

extend_tag:
 default_tag: "new_tag"
 tag_map:
 - tag: "my.app.sql"
 jmespath: "[?category=='sql']"
 - tag: "my.app.eh.storage"
 jmespath: "[?category=='storage']"
 ...
 jmespath_refs:
 jmespath_ref_1: "{jmespath_expression_1}"
 jmespath_ref_2: "{jmespath_expression_2}"
 ...

"extend_tag": {
 "default_tag": "new_tag",
 "tag_map": [
 {
 "tag": "my.app.sql",
 "jmespath": "[?category=='sql']"
 },
 {
 "tag": "my.app.eh.storage",
 "jmespath": "[?category=='storage']"
 }
 ........
 ],
 "jmespath_refs": {
 "jmespath_ref_1": "{jmespath_expression_1}",
 "jmespath_ref_2": "{jmespath_expression_2}"
 ........
 }
 }

If the original, internal tag structure looks like this:

tag:
 default_tag: "my.app.eh"
 tag_map:
 - tag: "my.app.eh.authentication"
 jmespath: "[?category=='auth']"
 - tag: "my.app.eh.sql"
 jmespath: "[?category=='sql']"

"tag": {
 "default_tag": "my.app.eh",
 "tag_map": [
 {
 "tag": "my.app.eh.authentication",
 "jmespath": "[?category=='auth']"
 },
 {
 "tag": "my.app.eh.sql",
 "jmespath": "[?category=='sql']"
 }
 ]
 }

And the extend_tag configuration is applied, the resultant tag will be:

tag:
 default_tag: "new_tag"
 tag_map:
 - tag: "my.app.eh.sql"
 jmespath: "[?category=='sql']"
 - tag: "my.app.eh.storage"
 jmespath: "[?category=='storage']"
 - tag: "my.app.eh.authentication"
 jmespath: "[?category=='auth']"
 jmespath_refs:
 jmespath_ref_1: "{jmespath_expression_1}"
 jmespath_ref_2: "{jmespath_expression_2}"

"tag": {
 "default_tag": "new_tag",
 "tag_map": [
 {
 "tag": "my.app.eh.sql",
 "jmespath": "[?category=='sql']"
 },
 {
 "tag": "my.app.eh.storage",
 "jmespath": "[?category=='storage']"
 },
 {
 "tag": "my.app.eh.authentication",
 "jmespath": "[?category=='auth']"
 }
 ],
 "jmespath_refs": {
 "jmespath_ref_1": "{jmespath_expression_1}",
 "jmespath_ref_2": "{jmespath_expression_2}"
 }
 }

On Premise

The collector should not be run on premise.

Structure

The following directory structure will be required as part of the setup procedure (it can be created under any directory):

<any_directory>
└── devo-collectors/
    └── azure/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config-azure.yaml

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any_directory>/devo-collectors/azure/certs. Learn more about security credentials in Devo here.

Editing the config.yaml file

In the config-azure.yaml file, replace the <app_id> , <active_directory_id>, <subscription_id> and <secret> values and enter the ones that you got in the previous steps. In the <short_unique_identifier> placeholder, enter the value that you choose.

globals:
  debug: false
  id: <collector_id_value>
  name: <collector_name_value>
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  azure:
    id: <short_unique_id>
    enabled: true
    credentials:
      subscription_id: <subscription_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
      tenant_id: <tenant_id_value>
    environment: <environment_value>
    services:
      vm_metrics:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        include_resource_id_patterns: [<include_resource_id_patterns_values>]
        exclude_resource_id_patterns: [<exclude_resource_id_patterns_values>]
  azure_event_hub:
    id: <short_unique_id>
    enabled: true
    credentials:
      subscription_id: <subscription_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
      tenant_id: <tenant_id_value>
    environment: <environment_value>
    services:
      event_hubs:
        override_pull_report_frequency_seconds: <override_pull_report_frequency_seconds_value>
        override_consumer_client_ttl_seconds: <override_consumer_client_ttl_seconds_value>
        queues:
          <queue_name_value>:
            namespace: <namespace_value>
            event_hub_name: <event_hub_name_value>
            event_hub_connection_string: <event_hub_connection_string_value>
            consumer_group: <consumer_group_value>
            events_use_auto_category: <events_use_auto_category_value>
            blob_storage_connection_string: <blob_storage_connection_string_value>
            blob_storage_container_name: <blob_storage_container_name_value>
            blob_storage_account_name: <blob_storage_account_name_value>
            compatibility_version: <compatibility_version_value>
            duplicated_messages_mechanism: <duplicated_messages_mechanism_value>
            override_starting_position: <override_starting_position_value>
            override_tag: <override_tag_value>
            client_thread_limit: <client_thread_limit_value>
            uamqp_transport: <uamqp_transport_value>
            partition_ids: [<partition_id>]
      event_hubs_auto_discover:
        resource_group: <resource_group_value>
        namespace: <namespace_value>
        blob_storage_account_name: <blob_storage_account_name_value>
        blob_storage_connection_string: <blob_storage_connection_string_value>
        consumer_group: <consumer_group_value>
        events_use_auto_category: <events_use_auto_category_value>
        duplicated_messages_mechanism: <duplicated_messages_mechanism_value>
        override_pull_report_frequency_seconds: <override_pull_report_frequency_seconds_value>
        override_consumer_client_ttl_seconds: <override_consumer_client_ttl_seconds_value>
        override_starting_position: <override_starting_position_value>
        override_blob_storage_container_prefix: <override_blob_storage_container_prefix_value>
        client_thread_limit: <client_thread_limit_value>
        uamqp_transport: <uamqp_transport_value>

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-azure_collector-docker-image-2.4.0	`ba1c93ed221b045ef5fe837567dd7305cca4b13dd748b3c6b2fce5053f21930b`

Use the following command to add the Docker image to the system:

gunzip -c collector-azure-docker-image-<version>.tgz | docker load

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/azure/

docker run \
--name collector-azure \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-azure.yaml \
--rm -it docker.devo.internal/collector/azure:<version>

Replace <version> with the corresponding value.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/azure/ directory.

version: '3'
services:
  collector-azure:
    image: docker.devo.internal/collector/azure:${IMAGE_VERSION:-latest}
    container_name: collector-azure
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-azure.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/azure/ directory:

IMAGE_VERSION=<version> docker-compose up -d

Replace <version> with the corresponding value.