Microsoft Azure collector migration guide (1.x.x to 2.0.0)

[ 1 Overview ] [ 2 Main updates ] [ 3 Preparing for migration ] [ 4 Migration steps ] [ 5 Troubleshooting ]

Overview

This guide will walk you through the process of updating your Microsoft Azure collector configuration from version 1.x.x to version 2.0.0. This new version introduces significant improvements and changes to the configuration style to enhance performance, usability, and security.

Main updates

The new configuration format introduces several key changes:

Multiple inputs: The configuration now supports multiple inputs to better represent the different data sources and access mechanisms (azure and azure_event_hub)
Credential config parameters renamed: The credentials configuration field names now follow names that are consistent with Microsoft Azure documentation: tenant_id, client_id, client_secret.
Azure Blob Storage checkpoint support: The configuration now accepts Azure Blob Storage related parameters in the queue-specific configuration: blob_storage_connection_string, blob_storage_container_name, blob_storage_account_name.
Moved VM Metrics to dedicated service: The VM metrics input has been moved to a dedicated service. The customer service configuration is no longer valid.
Moved Event Hub to dedicated service: The Event Hub input has been moved to a dedicated service. The customer service configuration is no longer valid.

Preparing for migration

Before starting the migration process, we recommend the following steps:

Backup your current configuration: Always ensure you have a backup of your existing configuration files to prevent any data loss.
Review the new configuration documentation: Familiarize yourself with the new configuration options available in version 2.0.0.

Migration steps

Step 1: Update credential configuration parameter field names

The credential configuration field names have been updated:

active_directory_id → tenant_id
secret → client_secret
app_id → client_id

An example of the old and new configuration is shown below:

# Old version
credentials:
  app_id: <app_id_value>
  active_directory_id: <active_directory_id_value>
  subscription_id: <subscription_id_value>
  secret: <secret_value>

↓

# New version
credentials:
  client_id: <client_id_value>
  tenant_id: <tenant_id_value>
  subscription_id: <subscription_id_value>
  client_secret: <client_secret_value>

Step 2: Update VM Metrics configuration

The VM Metrics service has been moved to the azure input and a dedicated vm_metrics service.

An example of the new configuration is shown below:

azure:
  id: <short_id>
  enabled: true
  credentials:
    client_id: <client_id_value>
    client_secret: <client_secret_value>
    tenant_id: <tenant_id_value>
  environment: <environment_value>
  services:
    vm_metrics:
      start_time_in_utc: <start_time_in_utc_value>
      request_period_in_seconds: 300

If you wish to continue from the old configuration, you must input the time of the latest in Devo in the start_time_in_utc field to indicate the time from which the puller will start collecting data.

Step 3: Update Event Hub configuration

The Event Hub service(s) have been moved to the azure_event_hub input and a dedicated event_hub service.

The new configuration now accepts the blob_storage_connection_string, blob_storage_container_name, and blob_storage_account_name parameters in the queue-specific configuration. These parameters are new, optional, and only required for those users who wish to leverage the Azure Blob Storage for checkpoint. This guide focuses on migrating the configuration from the old version to the new version -- for this reason, the new Azure Blob Storage checkpoint parameters are not relevant to older configurations because they use local, file-based checkpointing.

By default, the collector will begin pulling from the latest event in the queue if there is not already a pre-existing checkpoint. To ensure your migrated collectors fetch from the last event previously sent to Devo, identify the datetime of the last event in Devo for the relevant queue and input it into the override_starting_position field in the format "%Y-%m-%dT%H:%M:%SZ". When the collector begins pulling from the queue, the collector will begin fetching from the indicated datetime for the first checkpoint.

Step 4: Example before and after configuration

Putting it all together, see below for an example of the old and new configuration:

↓

Troubleshooting

Common Issue 1: Description and solution.
Common Issue 2: Description and solution.

For more detailed troubleshooting, refer to the official documentation or contact support.