Overview
This guide will walk you through the process of updating your Microsoft Azure collector configuration from version 1.7.1 to version 1.8.0. This new version introduces significant improvements and changes to the configuration style to enhance performance, usability, and security.
Main updates
The new configuration format introduces several key changes:
Multiple inputs: The configuration now supports multiple inputs to better represent the different data sources and access mechanisms (
azureandazure_event_hub)Credential config parameters renamed: The credentials configuration field names now follow names that are consistent with Microsoft Azure documentation:
tenant_id,client_id,client_secret.Azure Blob Storage checkpoint support: The configuration now accepts Azure Blob Storage related parameters in the queue-specific configuration:
blob_storage_connection_string,blob_storage_container_name,blob_storage_account_name.Moved VM Metrics to dedicated service: The VM metrics input has been moved to a dedicated service. The customer service configuration is no longer valid.
Moved Event Hub to dedicated service: The Event Hub input has been moved to a dedicated service. The customer service configuration is no longer valid.
Preparing for migration
Before starting the migration process, we recommend the following steps:
Backup your current configuration: Always ensure you have a backup of your existing configuration files to prevent any data loss.
Review the new configuration documentation: Familiarize yourself with the new configuration options available in version 1.8.0.
Migration steps
Step 1: Update credential configuration parameter field names
The credential configuration field names have been updated:
active_directory_id→tenant_idsecret→client_secretapp_id→client_id
An example of the old and new configuration is shown below:
# Old version (1.7.1) credentials: app_id: <app_id_value> active_directory_id: <active_directory_id_value> subscription_id: <subscription_id_value> secret: <secret_value>
↓
# New version (1.8.0) credentials: client_id: <client_id_value> tenant_id: <tenant_id_value> subscription_id: <subscription_id_value> client_secret: <client_secret_value>
Step 2: Update VM Metrics configuration
The VM Metrics service has been moved to the azure input and a dedicated vm_metrics service.
An example of the new configuration is shown below:
azure:
id: <short_id>
enabled: true
credentials:
client_id: <client_id_value>
client_secret: <client_secret_value>
tenant_id: <tenant_id_value>
environment: <environment_value>
services:
vm_metrics:
start_time_in_utc: <start_time_in_utc_value>
request_period_in_seconds: 300
If you wish to continue from the old configuration, you must input the time of the latest in Devo in the start_time_in_utc field to indicate the time from which the puller will start collecting data.
Step 3: Update Event Hub configuration
The Event Hub service(s) have been moved to the azure_event_hub input and a dedicated event_hub service.
azure_event_hub:
id: <short_id>
enabled: true
credentials:
client_id: <client_id_value>
client_secret: <client_secret_value>
tenant_id: <tenant_id_value>
environment: <environment_value>
services:
event_hubs:
queues:
<queue_name>:
event_hub_name: <event_hub_name_value>
event_hub_connection_string: <event_hub_connection_string_value>
consumer_group: <consumer_group_value>
events_use_autocategory: <events_use_autocategory_value>
blob_storage_connection_string: <blob_storage_connection_string_value>
blob_storage_container_name: <blob_storage_container_name_value>
blob_storage_account_name: <blob_storage_account_name_value>
compatibility_version: <compatibility_version_value>
duplicated_messages_mechanism: <duplicated_messages_mechanism>
override_starting_position: <override_starting_position_value>
The new configuration now accepts the blob_storage_connection_string, blob_storage_container_name, and blob_storage_account_name parameters in the queue-specific configuration. These parameters are new, optional, and only required for those users who wish to leverage the Azure Blob Storage for checkpoint. This guide focuses on migrating the configuration from the old version to the new version -- for this reason, the new Azure Blob Storage checkpoint parameters are not relevant to older configurations because they use local, file-based checkpointing.
By default, the collector will begin pulling from the latest event in the queue if there is not already a pre-existing checkpoint. To ensure your migrated collectors fetch from the last event previously sent to Devo, identify the datetime of the last event in Devo for the relevant queue and input it into the override_starting_position field in the format "%Y-%m-%dT%H:%M:%SZ". When the collector begins pulling from the queue, the collector will begin fetching from the indicated datetime for the first checkpoint.
Step 4: Example before and after configuration
Putting it all together, see below for an example of the old and new configuration:
# Old version (1.7.1)
inputs:
azure:
id: 10001
enabled: true
credentials:
app_id: app_id_acme
active_directory_id: active_directory_id_acme
subscription_id: subscription_id_acme
secret: secret_acme
environment: test_environment
requests_limits:
- period: 1d
number_of_requests: -1
services:
my_service_1:
request_period_in_seconds: 300
types:
- eh_services
queues:
queue_a:
event_hub_name: the-event-hub-name
consumer_group: the-consumer-group
connection_str: the-connection-string
events_use_autocategory: true
compatibility_version: 1.2.1
duplicated_messages_mechanism: global
use_global_counter_per_queue: true
my_service_2:
request_period_in_seconds: 300
types:
- vm_metrics
↓
# New version (1.8.0)
inputs:
azure:
id: 100001
enabled: true
credentials:
subscription_id: subscription_id_acme
client_id: app_id_acme
client_secret: secret_acme
tenant_id: active_directory_id_acme
environment: test-env
services:
vm_metrics:
request_period_in_seconds: 300
azure_event_hubs:
id: 100001
enabled: true
credentials:
subscription_id: subscription_id_acme
client_id: app_id_acme
client_secret: secret_acme
tenant_id: active_directory_id_acme
environment: test-env
services:
event_hubs:
queues:
queue_a:
event_hub_name: the-event-hub-name
event_hub_connection_string: the-connection-string
consumer_group: the-consumer-group
events_use_autocategory: true
compatibility_version: 1.2.0
duplicated_messages_mechanism: global
override_starting_position: "2022-01-01T00:00:00Z" # Replace with the datetime of the last event in Devo. Otherwise, collector pulls from latest event for the first checkpoint.
Troubleshooting
Common Issue 1: Description and solution.
Common Issue 2: Description and solution.
For more detailed troubleshooting, refer to the official documentation or contact support.