Content Comparison

Table of Contents

minLevel	1
maxLevel	2
outline	false
type	flat
separator	brackets
printable	true

...

Purpose

The Microsoft Azure is an ever-expanding set of collector gets data from Azure cloud computing services to help your organization meet its business challenges. Azure gives you the freedom to build, manage, and deploy applications on a massive, global network using your preferred tools and frameworks.

1.x to 2.x migrating guide

If you are migrating from v1.x to v2.x, you can find a complete guide in this article.. Common uses are:

Detect malicious Active Directory authentication
Detect malicious role, policy, and group changes
Correlate risky users identified by Entra ID with data you have in Devo
Detect malicious Application Gateway traffic
Detect failures and measure costs of virtual machines

Devo collector features

Features

Details

Allow parallel downloading (multipod)

Partial (supported for event_hubs services using Azure Blob Storage)

Note
The vm_metrics service cannot work in multipod mode. If you want to use the event_hubs service in multipod mode, you must not include a vm_service in the same collector.

Running environments

collector server
on-premise

Populated Devo events

table

Flattening pre-processing

no

Allowed source events obfuscation

yes

Data source description

Data source

Description

API endpoint

Collector service name

Devo table

VM Metrics

With the advantages of the Microsoft Azure API, one can obtain metrics about the deployed Virtual Machines, gathering them on our platform, making it easier to query and analyze in the Devo platform and Activeboards.

Azure Compute Management Client SDK and Azure Monitor Management Client SDK

vm_metrics

cloud.azure.vm.metrics_simple

Event Hubs

Several Microsoft Azure services can generate some type of execution information to be sent to an EventHub service. (see next section)

Azure Event Hubs SDK

event_hubs and event_hubs_autodiscover

<auto_tag_description>

Info
Valid for all cloud.azure tables by setting the output option to stream to Event Hub.

...

Rw ui tabs macro

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

In the Collector Server GUI, access the domain in which you want this instance to be created
Click Add Collector and find the one you wish to add.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.
In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

Code Block

{
 "global_overrides": {
 "debug": false
 },
 "inputs": {
 "azure": {
 "id": "<short_unique_id>",
 "enabled": true,
 "credentials": {
 "subscription_id": "<subscription_id_value>",
 "client_id": "<client_id_value>",
 "client_secret": "<client_secret_value>",
 "tenant_id": "<tenant_id_value>"
 },
 "environment": "<environment_value>",
 "services": {
 "vm_metrics": {
 "request_period_in_seconds": "<request_period_in_seconds_value>",
 "start_time_in_utc": "<start_time_in_utc_value>",
 "include_resource_id_patterns": [
 "<include_resource_id_patterns_values>"
 ],
 "exclude_resource_id_patterns": [
 "<exclude_resource_id_patterns_values>"
 ]
 }
 }
 },
 "azure_event_hub": {
 "id": "<short_unique_id>",
 "enabled": true,
 "credentials": {
 "subscription_id": "<subscription_id_value>",
 "client_id": "<client_id_value>",
 "client_secret": "<client_secret_value>",
 "tenant_id": "<tenant_id_value>"
 },
 "environment": "<environment_value>",
 "services": {
 "event_hubs": {
 "override_pull_report_frequency_seconds": "<override_pull_report_frequency_seconds_value>",
 "override_consumer_client_ttl_seconds": "<override_consumer_client_ttl_seconds_value>",
 "queues": {
 "<queue_name_value>": {
 "namespace": "<namespace_value>",
 "event_hub_name": "<event_hub_name_value>",
 "event_hub_connection_string": "<event_hub_connection_string_value>",
 "consumer_group": "<consumer_group_value>",
 "blob_storage_connection_string": "<blob_storage_connection_string_value>",
 "blob_storage_container_name": "<blob_storage_container_name_value>",
 "blob_storage_account_name": "<blob_storage_account_name_value>",
 "compatibility_version": "<compatibility_version_value>",
 "duplicated_messages_mechanism": "<duplicated_messages_mechanism_value>",
 "override_starting_position": "<override_starting_position_value>",
 "override_tag": "<override_tag_value>",
 "extend_tag": "<extend_tag_value>",
 "client_thread_limit": "<client_thread_limit_value>",
 "uamqp_transport": "<uamqp_transport_value>",
 "partition_ids": ["<partition_id>"]
 }
 }
 },
 "event_hubs_auto_discover": {
 "resource_group": "<resource_group_value>",
 "namespace": "<namespace_value>",
 "blob_storage_account_name": "<blob_storage_account_name_value>",
 "blob_storage_connection_string": "<blob_storage_connection_string_value>",
 "consumer_group": "<consumer_group_value>",
 "duplicated_messages_mechanism": "<duplicated_messages_mechanism_value>",
 "override_pull_report_frequency_seconds": "<override_pull_report_frequency_seconds_value>",
 "override_consumer_client_ttl_seconds": "<override_consumer_client_ttl_seconds_value>",
 "override_starting_position": "<override_starting_position_value>",
 "override_blob_storage_container_prefix": "<override_blob_storage_container_prefix_value>",
 "client_thread_limit": "<client_thread_limit_value>",
 "uamqp_transport": "<uamqp_transport_value>"
 }
 }
 }
 }
}

The following table outlines the parameters available for configuring the collector. Each parameter is categorized by its necessity (mandatory or optional), data type, acceptable values or formats, and a brief description.

Parameter	Data type	Requirement	Value range / Format	Description
`short_unique_id`	`str`	Mandatory	Min length: 1, Max length: 5	Short, unique ID for input service, used in persistence addressing. Avoid duplicates to prevent collisions.
`tenant_id_value`	`str`	Mandatory	Min length: 1	Tenant ID for Azure authentication.
`client_id_value`	`str`	Mandatory	Min length: 1	Client ID for Azure authentication.
`client_secret_value`	`str`	Mandatory	Min length: 1	Client secret for Azure authentication.
`subscription_id_value`	`str`	Mandatory	Min length: 1	Azure subscription ID.
`environment_value`	`str`	Optional	Min length: 1	Differentiates environments (e.g., dev, prod). Remove if unused.
`request_period_in_seconds_value`	`int`	Optional	Min: 60	Custom period in seconds between data pulls, overriding default (300s).
`start_time_in_utc_value`	`str`	Optional	UTC datetime format: `%Y-%m-%dT%H-%M-%SZ`	Custom start date for data retrieval, for historical data download. Remove if unused.
`include_resource_id_patterns_values`	`[str]`	Optional	Glob patterns e.g., `["VM-GROUP-1"]`	Includes resources matching patterns. Remove if unused.
`exclude_resource_id_patterns_values`	`[str]`	Optional	Glob patterns e.g., `["VM-GROUP-1"]`	Excludes resources matching patterns. Remove if unused.
`queue_name_value`	`str`	Mandatory	Min length: 1	Name for the queue, appears in related logs.
`event_hub_name_value`	`str`	Mandatory	Min length: 1	Name of the Event Hub to pull events from.
`event_hub_connection_string_value`	`str`	Mandatory	Min length: 1	Connection string for the Event Hub.
`consumer_group_value`	`str`	Optional	Min length: 1, Default: `$Default`	Consumer group for the Event Hub. Defaults to `$Default`.
`events_use_autocategory_value`	`bool`	Optional	Default: `true`	Enables auto-tagging of events. This value is always true.
`blob_storage_connection_string_value`	`str`	Optional	Min length: 1	Connection string for blob storage, optional for Azure Blob Storage checkpointing.
`blob_storage_container_name_value`	`str`	Optional	Min length: 1	Blob storage container name, required if using Azure Blob Storage checkpointing.
`blob_storage_account_name_value`	`str`	Optional	Min length: 1	Blob storage account name, alternative to using connection string for checkpointing.
`compatibility_version_value`	`str`	Optional	Version strings	Compatibility version for event processing.
`duplicated_messages_mechanism_value`	`str`	Optional	One of: `"local"`, `"global"`, `"none"`	Deduplication mechanism for messages: local, global, or none.
`override_starting_position_value`	`str`	Optional	One of: `"-1"`, `"@latest"`, `"[UTC datetime value]"`	Starting position for event fetching: from the beginning of available data (-1), from the latest data fetched (@fetched), or a specific datetime (%Y-%m- %dT%H-%M-%SZ format).
`override_tag_value`	`str`	Optional	Tag-friendly string	Optional tag to override the default tagging mechanism. See Event Hubs Tagging Configuration.
`extend_tag_value`	`str`	Optional	Object that can include any of the following properties: default_tag, tag_map, jmespath_refs	Advanced feature. Allows users to add/update various properties of the tag. If the user utilized `override_tag` and configured a simple tag string, this parameter will have no effect. If supplied, `default_tag` overrides the default tag, `jmespath_refs` adds/updates jmespath substitution values, and `tag_map` will add/update various tag paths to the pre-existing tag map. See Event Hubs Tagging Configuration.
`override_pull_report_frequency_seconds_value`	`int`	Optional	Default: 60	Frequency in seconds for reporting pull statistics in logs.
`override_consumer_client_ttl_seconds_value`	`int`	Optional	Default varies by service	Time-to-live in seconds for consumer clients, after which the collector restarts the pull cycle.
`resource_group_value`	`str`	Mandatory	Min length: 1	Azure resource group for event hub discovery.
`namespace_value`	`str`	Mandatory	Min length: 1	Namespace within Azure for event hub discovery.
`override_blob_storage_container_prefix_value`	`str`	Optional	Min length: 3, Max length: 10; Default: `devo-`	Prefix for blob storage containers created by auto-discovery service. Remove if unused.
`uamqp_transport_value`	`bool`	Optional	Default: `false`	Allows users to override/force event hub SDK to use legacy UAMQP transport mechanism (true)instead of the default/current PyAMQP mechanism (false).
`<partition_ids>`	`str`	Optional	List of partition number, as `["1","3","5","7"]`	Allows to define which partitions are going to be connected by this instance of the collector. It overrides `client_thread_limit_value`
`client_thread_limit_value`	`int`	Optional	Min value: 1	Adv feature - most users should use `partition_ids` instead to explicitly define what partitions the collector instance will query. Number of consumer threads that the collector will create. By default, collector will create as many threads as there are consumers in the event hub.

Info
Parameters marked as "Mandatory" are required for the collector's configuration. Optional parameters can be omitted or removed if not used, but they provide additional customization and control over the collector's behavior.

Note

Local deduplication means that duplicates are deleted in the data received from the current collector. Global means that duplicates are search for all the instances of the collector. None means that duplicates are not deleted.

See more details in the section Internal Process and Deduplication Method within the Even Hubs section of the Collector Services Detail.

If you deploy one collector, use local. If you deploy several instances of the collector, use global.

Note
`override_tag_value` can be used to create new categories. If needed, consult the Event Hubs tagging Configuration within the Even Hubs section of the Collector Services Detail.

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure will be required as part of the setup procedure (it can be created under any directory):

Code Block

<any_directory>
└── devo-collectors/
    └── azure/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config-azure.yaml

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any_directory>/devo-collectors/azure/certs. Learn more about security credentials in Devo here.

Editing the config.yaml file

In the config-azure.yaml file, replace the <app_id> , <active_directory_id>, <subscription_id> and <secret> values and enter the ones that you got in the previous steps. In the <short_unique_identifier> placeholder, enter the value that you choose.

Code Block

language	yaml

globals:
  debug: false
  id: <collector_id_value>
  name: <collector_name_value>
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename>
      cert: <cert_filename>
      key: <key_filename>
inputs:
  azure:
    id: <short_unique_id>
    enabled: true
    credentials:
      subscription_id: <subscription_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
      tenant_id: <tenant_id_value>
    environment: <environment_value>
    services:
      vm_metrics:
        request_period_in_seconds: <request_period_in_seconds_value>
        start_time_in_utc: <start_time_in_utc_value>
        include_resource_id_patterns: [<include_resource_id_patterns_values>]
        exclude_resource_id_patterns: [<exclude_resource_id_patterns_values>]
  azure_event_hub:
    id: <short_unique_id>
    enabled: true
    credentials:
      subscription_id: <subscription_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
      tenant_id: <tenant_id_value>
    environment: <environment_value>
    services:
      event_hubs:
        override_pull_report_frequency_seconds: <override_pull_report_frequency_seconds_value>
        override_consumer_client_ttl_seconds: <override_consumer_client_ttl_seconds_value>
        queues:
          <queue_name_value>:
            namespace: <namespace_value>
            event_hub_name: <event_hub_name_value>
            event_hub_connection_string: <event_hub_connection_string_value>
            consumer_group: <consumer_group_value>
            events_use_auto_category: <events_use_auto_category_value>
            blob_storage_connection_string: <blob_storage_connection_string_value>
            blob_storage_container_name: <blob_storage_container_name_value>
            blob_storage_account_name: <blob_storage_account_name_value>
            compatibility_version: <compatibility_version_value>
            duplicated_messages_mechanism: <duplicated_messages_mechanism_value>
            override_starting_position: <override_starting_position_value>
            override_tag: <override_tag_value>
            client_thread_limit: <client_thread_limit_value>
            uamqp_transport: <uamqp_transport_value>
            partition_ids: [<partition_id>]
      event_hubs_auto_discover:
        resource_group: <resource_group_value>
        namespace: <namespace_value>
        blob_storage_account_name: <blob_storage_account_name_value>
        blob_storage_connection_string: <blob_storage_connection_string_value>
        consumer_group: <consumer_group_value>
        events_use_auto_category: <events_use_auto_category_value>
        duplicated_messages_mechanism: <duplicated_messages_mechanism_value>
        override_pull_report_frequency_seconds: <override_pull_report_frequency_seconds_value>
        override_consumer_client_ttl_seconds: <override_consumer_client_ttl_seconds_value>
        override_starting_position: <override_starting_position_value>
        override_blob_storage_container_prefix: <override_blob_storage_container_prefix_value>
        client_thread_limit: <client_thread_limit_value>
        uamqp_transport: <uamqp_transport_value>

Parameter	Data type	Requirement	Value range / Format	Description
`collector_id_value`	`str`	Mandatory	Min length: 1, Max length: 5	Unique identifier for the collector.
`collector_name_value`	`str`	Mandatory	Min length: 1, Max length: 10	Name assigned to the collector.
`devo_address`	`str`	Mandatory	One of: `collector-us.devo.io`, `collector-eu.devo.io`	Devo Cloud destination for events.
`chain_filename`	`str`	Mandatory	Min length: 4, Max length: 20	Filename of the `chain.crt` file from your Devo domain.
`cert_filename`	`str`	Mandatory	Min length: 4, Max length: 20	Filename of the `file.cert` from your Devo domain.
`key_filename`	`str`	Mandatory	Min length: 4, Max length: 20	Filename of the `file.key` from your Devo domain.
`short_unique_id`	`str`	Mandatory	Min length: 1, Max length: 5	Short, unique ID for input service, used in persistence addressing. Avoid duplicates to prevent collisions.
`tenant_id_value`	`str`	Mandatory	Min length: 1	Tenant ID for Azure authentication.
`client_id_value`	`str`	Mandatory	Min length: 1	Client ID for Azure authentication.
`client_secret_value`	`str`	Mandatory	Min length: 1	Client secret for Azure authentication.
`subscription_id_value`	`str`	Mandatory	Min length: 1	Azure subscription ID.
`environment_value`	`str`	Optional	Min length: 1	Differentiates environments (e.g., dev, prod). Remove if unused.
`request_period_in_seconds_value`	`int`	Optional	Min: 60	Custom period in seconds between data pulls, overriding default (300s).
`start_time_in_utc_value`	`str`	Optional	UTC datetime format: `%Y-%m-%dT%H-%M-%SZ`	Custom start date for data retrieval, for historical data download. Remove if unused.
`include_resource_id_patterns_values`	`[str]`	Optional	Glob patterns e.g., `["VM-GROUP-1"]`	Includes resources matching patterns. Remove if unused.
`exclude_resource_id_patterns_values`	`[str]`	Optional	Glob patterns e.g., `["VM-GROUP-1"]`	Excludes resources matching patterns. Remove if unused.
`queue_name_value`	`str`	Mandatory	Min length: 1	Name for the queue, appears in related logs.
`event_hub_name_value`	`str`	Mandatory	Min length: 1	Name of the Event Hub to pull events from.
`event_hub_connection_string_value`	`str`	Mandatory	Min length: 1	Connection string for the Event Hub.
`consumer_group_value`	`str`	Optional	Min length: 1, Default: `$Default`	Consumer group for the Event Hub. Defaults to `$Default`.
`events_use_autocategory_value`	`bool`	Optional	Default: `false`	Enables/disables auto-tagging of events.
`blob_storage_connection_string_value`	`str`	Optional	Min length: 1	Connection string for blob storage, optional for Azure Blob Storage checkpointing.
`blob_storage_container_name_value`	`str`	Optional	Min length: 1	Blob storage container name, required if using Azure Blob Storage checkpointing.
`blob_storage_account_name_value`	`str`	Optional	Min length: 1	Blob storage account name, alternative to using connection string for checkpointing.
`compatibility_version_value`	`str`	Optional	Version strings	Compatibility version for event processing.
`duplicated_messages_mechanism_value`	`str`	Optional	One of: `"local"`, `"global"`, `"none"`	Deduplication mechanism for messages: local, global, or none (see note below).
`override_starting_position_value`	`str`	Optional	One of: `"-1"`, `"@latest"`, `"[UTC datetime value]"`	Starting position for event fetching: from the beginning of available data (-1), from the latest data fetched (@fetched), or a specific datetime (%Y-%m- %dT%H-%M-%SZ format).
`override_tag_value`	`str`	Optional	Tag-friendly string	Optional tag to override the default tagging mechanism. See Event Hubs Tagging Configuration.
`extend_tag_value`	`str`	Optional	Object that can include any of the following properties: `default_tag`, `tag_map`, `jmespath_refs`.	Advanced feature. Allows users to add/update various properties of the tag. If the user utilized `override_tag` and configured a simple tag string, this parameter will have no effect. If supplied, `default_tag` overrides the default tag, `jmespath_refs` add/update jmespath substitution values, and `tag_map` will add/update various tag paths to the pre-existing tag map. See Event Hubs Tagging Configuration.
`override_pull_report_frequency_seconds_value`	`int`	Optional	Default: 60	Frequency in seconds for reporting pull statistics in logs.
`override_consumer_client_ttl_seconds_value`	`int`	Optional	Default varies by service	Time-to-live in seconds for consumer clients, after which the collector restarts the pull cycle.
`resource_group_value`	`str`	Mandatory	Min length: 1	Azure resource group for event hub discovery.
`namespace_value`	`str`	Mandatory	Min length: 1	Namespace within Azure for event hub discovery.
`override_blob_storage_container_prefix_value`	`str`	Optional	Min length: 3, Max length: 10; Default: `devo-`	Prefix for blob storage containers created by auto-discovery service. Remove if unused.
`uamqp_transport_value`	`bool`	Optional	Default: `false`	Allows users to override/force event hub SDK to use legacy UAMQP transport mechanism (true)instead of the default/current PyAMQP mechanism (false)
`<partition_ids>`	`str`	Optional	List of partition number, as `["1","3","5","7"]`	Allows to define which partitions are going to be connected by this instance of the collector. It overrides `client_thread_limit_value`
`client_thread_limit_value`	`int`	Optional	Min value: 1	Advanced feature - most users should use `partition_ids` instead to explicitly define what partitions the collector instance will query. Number of consumer threads that the collector will create. By default, collector will create as many threads as there are consumers in the event hub.

Info
Parameters marked as "Mandatory" are required for the collector's configuration. Optional parameters can be omitted or removed if not used, but they provide additional customization and control over the collector's behavior.

Note

Local deduplication means that duplicates are deleted in the data received from the current collector. Global means that duplicates are search for all the instances of the collector. None means that duplicates are not deleted.

See more details in the section Internal Process and Deduplication Method.

If you deploy one collector, use local. If you deploy several instances of the collector, use global.

Note
`override_tag_value` can be used to create new categories. If needed, consult the Event Hubs tagging Configuration within the Even Hubs section of the Collector Services Detail.

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-azure_collector-docker-image-2.4.0	`ba1c93ed221b045ef5fe837567dd7305cca4b13dd748b3c6b2fce5053f21930b`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c collector-azure-docker-image-<version>.tgz \| docker load

Info
Once the Docker image is imported, it will show the real name of the Docker image (including version info).

The Docker image can be deployed on the following services:

Anchor
docker
docker
Docker

Execute the following command on the root directory <any_directory>/devo-collectors/azure/

Code Block

docker run \
--name collector-azure \
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config-azure.yaml \
--rm -it docker.devo.internal/collector/azure:<version>

Note
Replace `<version>` with the corresponding value.

Anchor
dockercompose
dockercompose
Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/azure/ directory.

Code Block

language	yaml

version: '3'
services:
  collector-azure:
    image: docker.devo.internal/collector/azure:${IMAGE_VERSION:-latest}
    container_name: collector-azure
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-azure.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/azure/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<version>` with the corresponding value.

...

Version	Old Version 1	New Version 2
Changes made by	Laszlo Frazer	Laszlo Frazer
Saved on	Feb 04, 2025	Feb 04, 2025

Versions Compared

Key

Purpose

1.x to 2.x migrating guide

Devo collector features

Data source description

Editing the JSON configuration

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Anchor
docker
docker
Docker

Anchor
dockercompose
dockercompose
Docker Compose

Content Comparison

Versions Compared

Key

Purpose

1.x to 2.x migrating guide

Devo collector features

Data source description

Editing the JSON configuration

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

AnchordockerdockerDocker

AnchordockercomposedockercomposeDocker Compose

Anchor
docker
docker
Docker

Anchor
dockercompose
dockercompose
Docker Compose