| Rw tab |
|---|
| title | On-premise collector |
|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure will be required as part of the setup procedure (it can be created under any directory): | Code Block |
|---|
<any_directory>
└── devo-collectors/
└── azure/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── state/
└── config/
└── config-azure.yaml |
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any_directory>/devo-collectors/azure/certs. Learn more about security credentials in Devo here.  Editing the config-azure.yaml fileIn the config-azure.yaml file, replace the <app_id> , <active_directory_id>, <subscription_id> and <secret> values and enter the ones that you got in the previous steps. In the <short_unique_identifier> placeholder, enter the value that you choose. | Code Block |
|---|
| globals:
debug: false
id: <collector_id_value>
name: <collector_name_value>
persistence:
type: filesystem
config:
directory_name: state
outputs:
devo_1:
type: devo_platform
config:
address: <devo_address>
port: 443
type: SSL
chain: <chain_filename>
cert: <cert_filename>
key: <key_filename>
inputs:
azure:
id: <short_unique_id>
enabled: true
credentials:
subscription_id: <subscription_id_value>
client_id: <client_id_value>
client_secret: <client_secret_value>
tenant_id: <tenant_id_value>
environment: <environment_value>
services:
vm_metrics:
request_period_in_seconds: <request_period_in_seconds_value>
start_time_in_utc: <start_time_in_utc_value>
include_resource_id_patterns: [<include_resource_id_patterns_values>]
exclude_resource_id_patterns: [<exclude_resource_id_patterns_values>]
azure_event_hub:
id: <short_unique_id>
enabled: true
credentials:
subscription_id: <subscription_id_value>
client_id: <client_id_value>
client_secret: <client_secret_value>
tenant_id: <tenant_id_value>
environment: <environment_value>
services:
event_hubs:
override_pull_report_frequency_seconds: <override_pull_report_frequency_seconds_value>
override_consumer_client_ttl_seconds: <override_consumer_client_ttl_seconds_value>
queues:
<queue_name_value>:
namespace: <namespace_value>
event_hub_name: <event_hub_name_value>
event_hub_connection_string: <event_hub_connection_string_value>
consumer_group: <consumer_group_value>
events_use_auto_category: <events_use_auto_category_value>
blob_storage_connection_string: <blob_storage_connection_string_value>
blob_storage_container_name: <blob_storage_container_name_value>
blob_storage_account_name: <blob_storage_account_name_value>
compatibility_version: <compatibility_version_value>
duplicated_messages_mechanism: <duplicated_messages_mechanism_value>
override_starting_position: <override_starting_position_value>
override_tag: <override_tag_value>
client_thread_limit: <client_thread_limit_value>
uamqp_transport: <uamqp_transport_value>
partition_ids: [<partition_id>]
event_hubs_auto_discover:
resource_group: <resource_group_value>
namespace: <namespace_value>
blob_storage_account_name: <blob_storage_account_name_value>
blob_storage_connection_string: <blob_storage_connection_string_value>
consumer_group: <consumer_group_value>
events_use_auto_category: <events_use_auto_category_value>
duplicated_messages_mechanism: <duplicated_messages_mechanism_value>
override_pull_report_frequency_seconds: <override_pull_report_frequency_seconds_value>
override_consumer_client_ttl_seconds: <override_consumer_client_ttl_seconds_value>
override_starting_position: <override_starting_position_value>
override_blob_storage_container_prefix: <override_blob_storage_container_prefix_value>
client_thread_limit: <client_thread_limit_value>
uamqp_transport: <uamqp_transport_value> |
| Info |
|---|
The tag field is optional and is only available in the eh_services services type. |
| Note |
|---|
For compatibility reasons, the default value of the events_use_autocategory property is false |
| Note |
|---|
For new deployments, we recommend the following values: |
| Info |
|---|
If you need to use a custom tag for generated messages, it can be done using the property tag inside any queue name, next toevent_hub_nameorconnection_str. For example tag: my.app.azure.{service_name} |
Parameter | Data type | Requirement | Value range / Format | Description |
|---|
collector_id_value
| str
| Mandatory | Min length: 1, Max length: 5 | Unique identifier for the collector. | collector_name_value
| str
| Mandatory | Min length: 1, Max length: 10 | Name assigned to the collector. | devo_address
| str
| Mandatory | One of: collector-us.devo.io, collector-eu.devo.io | Devo Cloud destination for events. | chain_filename
| str
| Mandatory | Min length: 4, Max length: 20 | Filename of the chain.crt file from your Devo domain. | cert_filename
| str
| Mandatory | Min length: 4, Max length: 20 | Filename of the file.cert from your Devo domain. | key_filename
| str
| Mandatory | Min length: 4, Max length: 20 | Filename of the file.key from your Devo domain. | short_unique_id
| str
| Mandatory | Min length: 1, Max length: 5 | Short, unique ID for input service, used in persistence addressing. Avoid duplicates to prevent collisions. | tenant_id_value
| str
| Mandatory | Min length: 1 | Tenant ID for Azure authentication. | client_id_value
| str
| Mandatory | Min length: 1 | Client ID for Azure authentication. | client_secret_value
| str
| Mandatory | Min length: 1 | Client secret for Azure authentication. | subscription_id_value
| str
| Mandatory | Min length: 1 | Azure subscription ID. | environment_value
| str
| Optional | Min length: 1 | Differentiates environments (e.g., dev, prod). Remove if unused. | request_period_in_seconds_value
| int
| Optional | Min: 60 | Custom period in seconds between data pulls, overriding default (300s). | start_time_in_utc_value
| str
| Optional | UTC datetime format: %Y-%m-%dT%H-%M-%SZ | Custom start date for data retrieval, for historical data download. Remove if unused. | include_resource_id_patterns_values
| [str]
| Optional | Glob patterns e.g., ["*VM-GROUP-1*"] | Includes resources matching patterns. Remove if unused. | exclude_resource_id_patterns_values
| [str]
| Optional | Glob patterns e.g., ["*VM-GROUP-1*"] | Excludes resources matching patterns. Remove if unused. | queue_name_value
| str
| Mandatory | Min length: 1 | Name for the queue, appears in related logs. | event_hub_name_value
| str
| Mandatory | Min length: 1 | Name of the Event Hub to pull events from. | event_hub_connection_string_value
| str
| Mandatory | Min length: 1 | Connection string for the Event Hub. | consumer_group_value
| str
| Optional | Min length: 1, Default: $Default | Consumer group for the Event Hub. Defaults to $Default. | events_use_autocategory_value
| bool
| Optional | Default: false | Enables/disables auto-tagging of events. | blob_storage_connection_string_value
| str
| Optional | Min length: 1 | Connection string for blob storage, optional for Azure Blob Storage checkpointing. | blob_storage_container_name_value
| str
| Optional | Min length: 1 | Blob storage container name, required if using Azure Blob Storage checkpointing. | blob_storage_account_name_value
| str
| Optional | Min length: 1 | Blob storage account name, alternative to using connection string for checkpointing. | compatibility_version_value
| str
| Optional | Version strings | Compatibility version for event processing. | duplicated_messages_mechanism_value
| str
| Optional | One of: "local", "global", "none" | Deduplication mechanism for messages: local, global, or none (see note below). | override_starting_position_value
| str
| Optional | One of: "-1", "@latest", "[UTC datetime value]" | Starting position for event
fetching: from the beginning of
available data (-1), from the
latest data fetched (@fetched),
or a specific datetime (%Y-%m-
%dT%H-%M-%SZ format). | override_tag_value
| str
| Optional | Tag-friendly string | Optional tag to override the default tagging mechanism. Remove if unused. | override_pull_report_frequency_seconds_value
| int
| Optional | Default: 60 | Frequency in seconds for reporting pull statistics in logs. | override_consumer_client_ttl_seconds_value
| int
| Optional | Default varies by service | Time-to-live in seconds for consumer clients, after which the collector restarts the pull cycle. | resource_group_value
| str
| Mandatory | Min length: 1 | Azure resource group for event hub discovery. | namespace_value
| str
| Mandatory | Min length: 1 | Namespace within Azure for event hub discovery. | override_blob_storage_container_prefix_value
| str
| Optional | Min length: 3, Max length: 10; Default: devo- | Prefix for blob storage containers created by auto-discovery service. Remove if unused. | uamqp_transport_value
| bool
| Optional | Default: false | Allows users to override/force
event hub SDK to use legacy
UAMQP transport mechanism
(true)instead of the
default/current PyAMQP
mechanism (false) | <partition_ids>
| str
| Optional | List of
partition
number, as
["1","3","5","7"] | Allows to define which partitions are going to be connected by this instance of the collector. It overrides client_thread_limit_value |
| Info |
|---|
Parameters marked as "Mandatory" are required for the collector's configuration. Optional parameters can be omitted or removed if not used, but they provide additional customization and control over the collector's behavior. |
| Note |
|---|
Local deduplication means that duplicates are deleted in the data received from the current collector. Global means that duplicates are search for all the instances of the collector. None means that duplicates are not deleted. See more details in the section Internal Process and Deduplication Method. If you deploy one collector, use local. If you deploy several instances of the collector, use global. |
| Note |
|---|
override_tag_value can be used to create new categories. If needed, consult this section.
|
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table: Use the following command to add the Docker image to the system: | Code Block |
|---|
gunzip -c collector-azure-docker-image-<version>.tgz | docker load |
| Info |
|---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). |
The Docker image can be deployed on the following services: DockerExecute the following command on the root directory <any_directory>/devo-collectors/azure/ | Code Block |
|---|
docker run \
--name collector-azure \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-azure.yaml \
--rm -it docker.devo.internal/collector/azure:<version> |
| Note |
|---|
Replace <version> with the corresponding value. |
| Anchor |
|---|
| dockercompose |
|---|
| dockercompose |
|---|
|
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/azure/ directory. | Code Block |
|---|
| version: '3'
services:
collector-azure:
image: docker.devo.internal/collector/azure:${IMAGE_VERSION:-latest}
container_name: collector-azure
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config-azure.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/azure/ directory: | Code Block |
|---|
IMAGE_VERSION=<version> docker-compose up -d |
| Note |
|---|
Replace <version> with the corresponding value. |
We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer: In the Collector Server GUI, access the domain in which you want this instance to be created Click Add Collector and find the one you wish to add. In the Version field, select the latest value. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain). In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups. In the Parameters section, establish the Collector Parameters as follows below:
Editing the JSON configuration| Code Block |
|---|
{
"global_overrides": {
"debug": false
},
"inputs": {
"azure": {
"id": "<short_unique_id>",
"enabled": true,
"credentials": {
"subscription_id": "<subscription_id_value>",
"client_id": "<client_id_value>",
"client_secret": "<client_secret_value>",
"tenant_id": "<tenant_id_value>"
},
"environment": "<environment_value>",
"services": {
"vm_metrics": {
"request_period_in_seconds": "<request_period_in_seconds_value>",
"start_time_in_utc": "<start_time_in_utc_value>",
"include_resource_id_patterns": [
"<include_resource_id_patterns_values>"
],
"exclude_resource_id_patterns": [
"<exclude_resource_id_patterns_values>"
]
}
}
},
"azure_event_hub": {
"id": "<short_unique_id>",
"enabled": true,
"credentials": {
"subscription_id": "<subscription_id_value>",
"client_id": "<client_id_value>",
"client_secret": "<client_secret_value>",
"tenant_id": "<tenant_id_value>"
},
"environment": "<environment_value>",
"services": {
"event_hubs": {
"override_pull_report_frequency_seconds": "<override_pull_report_frequency_seconds_value>",
"override_consumer_client_ttl_seconds": "<override_consumer_client_ttl_seconds_value>",
"queues": {
"<queue_name_value>": {
"namespace": "<namespace_value>",
"event_hub_name": "<event_hub_name_value>",
"event_hub_connection_string": "<event_hub_connection_string_value>",
"consumer_group": "<consumer_group_value>",
"events_use_auto_category": "<events_use_auto_category_value>",
"blob_storage_connection_string": "<blob_storage_connection_string_value>",
"blob_storage_container_name": "<blob_storage_container_name_value>",
"blob_storage_account_name": "<blob_storage_account_name_value>",
"compatibility_version": "<compatibility_version_value>",
"duplicated_messages_mechanism": "<duplicated_messages_mechanism_value>",
"override_starting_position": "<override_starting_position_value>",
"override_tag": "<override_tag_value>",
"client_thread_limit": "<client_thread_limit_value>",
"uamqp_transport": "<uamqp_transport_value>",
"partition_ids": ["<partition_id>"]
}
}
},
"event_hubs_auto_discover": {
"resource_group": "<resource_group_value>",
"namespace": "<namespace_value>",
"blob_storage_account_name": "<blob_storage_account_name_value>",
"blob_storage_connection_string": "<blob_storage_connection_string_value>",
"consumer_group": "<consumer_group_value>",
"events_use_auto_category": "<events_use_auto_category_value>",
"duplicated_messages_mechanism": "<duplicated_messages_mechanism_value>",
"override_pull_report_frequency_seconds": "<override_pull_report_frequency_seconds_value>",
"override_consumer_client_ttl_seconds": "<override_consumer_client_ttl_seconds_value>",
"override_starting_position": "<override_starting_position_value>",
"override_blob_storage_container_prefix": "<override_blob_storage_container_prefix_value>",
"client_thread_limit": "<client_thread_limit_value>",
"uamqp_transport": "<uamqp_transport_value>"
}
}
}
}
} |
The following table outlines the parameters available for configuring the collector. Each parameter is categorized by its necessity (mandatory or optional), data type, acceptable values or formats, and a brief description. Parameter | Data type | Requirement | Value range / Format | Description |
|---|
short_unique_id
| str
| Mandatory | Min length: 1, Max length: 5 | Short, unique ID for input service, used in persistence addressing. Avoid duplicates to prevent collisions. | tenant_id_value
| str
| Mandatory | Min length: 1 | Tenant ID for Azure authentication. | client_id_value
| str
| Mandatory | Min length: 1 | Client ID for Azure authentication. | client_secret_value
| str
| Mandatory | Min length: 1 | Client secret for Azure authentication. | subscription_id_value
| str
| Mandatory | Min length: 1 | Azure subscription ID. | environment_value
| str
| Optional | Min length: 1 | Differentiates environments (e.g., dev, prod). Remove if unused. | request_period_in_seconds_value
| int
| Optional | Min: 60 | Custom period in seconds between data pulls, overriding default (300s). | start_time_in_utc_value
| str
| Optional | UTC datetime format: %Y-%m-%dT%H-%M-%SZ | Custom start date for data retrieval, for historical data download. Remove if unused. | include_resource_id_patterns_values
| [str]
| Optional | Glob patterns e.g., ["*VM-GROUP-1*"] | Includes resources matching patterns. Remove if unused. | exclude_resource_id_patterns_values
| [str]
| Optional | Glob patterns e.g., ["*VM-GROUP-1*"] | Excludes resources matching patterns. Remove if unused. | queue_name_value
| str
| Mandatory | Min length: 1 | Name for the queue, appears in related logs. | event_hub_name_value
| str
| Mandatory | Min length: 1 | Name of the Event Hub to pull events from. | event_hub_connection_string_value
| str
| Mandatory | Min length: 1 | Connection string for the Event Hub. | consumer_group_value
| str
| Optional | Min length: 1, Default: $Default | Consumer group for the Event Hub. Defaults to $Default. | events_use_autocategory_value
| bool
| Optional | Default: false | Enables/disables auto-tagging of events. | blob_storage_connection_string_value
| str
| Optional | Min length: 1 | Connection string for blob storage, optional for Azure Blob Storage checkpointing. | blob_storage_container_name_value
| str
| Optional | Min length: 1 | Blob storage container name, required if using Azure Blob Storage checkpointing. | blob_storage_account_name_value
| str
| Optional | Min length: 1 | Blob storage account name, alternative to using connection string for checkpointing. | compatibility_version_value
| str
| Optional | Version strings | Compatibility version for event processing. | duplicated_messages_mechanism_value
| str
| Optional | One of: "local", "global", "none" | Deduplication mechanism for messages: local, global, or none. | override_starting_position_value
| str
| Optional | One of: "-1", "@latest", "[UTC datetime value]" | Starting position for event
fetching: from the beginning of
available data (-1), from the
latest data fetched (@fetched),
or a specific datetime (%Y-%m-
%dT%H-%M-%SZ format). | override_tag_value
| str
| Optional | Tag-friendly string | Optional tag to override the default tagging mechanism. Remove if unused. | override_pull_report_frequency_seconds_value
| int
| Optional | Default: 60 | Frequency in seconds for reporting pull statistics in logs. | override_consumer_client_ttl_seconds_value
| int
| Optional | Default varies by service | Time-to-live in seconds for consumer clients, after which the collector restarts the pull cycle. | resource_group_value
| str
| Mandatory | Min length: 1 | Azure resource group for event hub discovery. | namespace_value
| str
| Mandatory | Min length: 1 | Namespace within Azure for event hub discovery. | override_blob_storage_container_prefix_value
| str
| Optional | Min length: 3, Max length: 10; Default: devo- | Prefix for blob storage containers created by auto-discovery service. Remove if unused. | uamqp_transport_value
| bool
| Optional | Default: false | Allows users to override/force
event hub SDK to use legacy
UAMQP transport mechanism
(true)instead of the
default/current PyAMQP
mechanism (false) | <partition_ids>
| str
| Optional | List of
partition
number, as
["1","3","5","7"] | Allows to define which partitions are going to be connected by this instance of the collector. It overrides client_thread_limit_value |
| Info |
|---|
Parameters marked as "Mandatory" are required for the collector's configuration. Optional parameters can be omitted or removed if not used, but they provide additional customization and control over the collector's behavior. |
| Note |
|---|
Local deduplication means that duplicates are deleted in the data received from the current collector. Global means that duplicates are search for all the instances of the collector. None means that duplicates are not deleted. See more details in the section Internal Process and Deduplication Method. If you deploy one collector, use local. If you deploy several instances of the collector, use global. |
| Note |
|---|
override_tag_value can be used to create new categories. If needed, consult this section.
|
|
