Microsoft Azure collector
- Juan Tomás Alonso Nieto (Deactivated)
Service description
Microsoft Azure is an ever-expanding set of cloud computing services to help your organization meet its business challenges. Azure gives you the freedom to build, manage, and deploy applications on a massive, global network using your preferred tools and frameworks.
Data source description
You can use the Microsoft Azure collector to send the following types of data to your Devo domain. Once the gathered information arrives at Devo, it will be processed and included in different tables in the associated Devo domain.
Virtual Machine metrics
With the advantages of the Microsoft Azure API, one can obtain metrics about the deployed Virtual Machines, gathering them on our platform, making it easier to query and analyze in the Devo platform and Activeboards.
Virtual Machine metric events are sent to the cloud.azure.vm.metrics_simple Devo table.
Event Hub Services
Many of the available Microsoft Azure services can generate some type of execution information to be sent to an EventHub service. This type of data can be categorized as events or metrics. The events, in turn, can be from different subtypes: audits, status, logs, etc.
All such data will be gathered by Devo’s Microsoft Azure collector and sent to our platform, where message auto-categorization functionality is enabled for sending the messages to relevant Devo tables in an automatic way.
Although EventHub is the service used for centralizing Azure services' data, it also generates information that can be sent to itself.
In case the amount of egress data exceeds Throughput per Unit limits set by Azure (2 MB/s or 4096 events per second), it won’t be possible for Devo to continue reliable ingestion of data. You can monitor ingress/egress throughput in Azure Portal EventHub Namespace, and based on trends/alerts, you can add another EventHub to resolve this. To avoid this from happening in the first place, please follow scalability guidance provided by Microsoft in their technical documentation.
Auto-categorization of Microsoft Azure service messages
In the table below are listed the patterns that will be used for detecting the message type, the Provider, Service, and Category pattern values would be used to route the message to the proper Devo table.
Each message stored in an EventHub service is generated by one data Provider and also by one Service, and finally, it's also having a Category field, which all together determine the message type.
Over time, the auto-categorization patterns have been improved and expanded in the different collector versions, the tables below contain the pattern values released in each version.
The collector versions not mentioned here are having changes not related to event mapping for auto-categorization functionality.
Setup
The Microsoft Azure collector centralizes the data with an Event Hub using the Azure SDK. To use it, you need to configure the resources in the Azure Portal and set the right permissions to access the information.
Virtual Machine metrics
Getting credentials
To log in to the Azure subscription, the collector uses a Service Principal object. You need to get the subscription ID, Active Directory ID, Application ID (service principal identification), and the client secret (service principal "password"). To get them, follow these steps:
Log in to your Azure account and search for Azure Active Directory.
Now, click App registrations in the left menu and click the app (or Service Principal) that you are going to use.
In the Overview area, find the Application (client) ID and the Directory (tenant) ID.
Now click Certificates & Secrets on the menu and create a new client secret by clicking the New client secret button.
Don't forget to save the client secret value, it will be only shown upon creation.Get the subscription ID by searching for Subscriptions on the home page.
Find the correct subscription and note down the subscription ID.
Setting up permissions
After creating the App registration (or Service Principal), go to the desired Resource Group (or subscription if you want to retrieve metrics from all the available virtual machines).
Select Access control (IAM) in the left menu and click Add.
Select at least the Reader role and choose the previously created App registration.
Confirm the changes.
Event Hub events
Getting credentials
In your Azure account, search for the Event Hubs service and click on it.
Create an Event Hub resource per region (repeat the steps below for each region):
Click Add.
Fill the mandatory fields keeping in mind that the Event Hub must be in the same region as the resources that you are going to monitor (and only need one per region). The Throughput Units option refers to the ingress/egress limit in MB/s (each unit is 1 MB/s or 1000 events/second ingress, 2 MB/s, or 4096 events/second egress). You should adjust it according to the data volume (this can be modified later).
The previous steps create an EventHub namespace; now go to Event Hubs, search the created one and click on it.
Now click on the + Event Hub button and create a new resource. You only need to fill the Name and Partition Count fields (the Partition Count field will divide the data into different partitions to make it easier to read large volumes of data). Write down the EventHub name to be used later in the configuration file.
Once the Event Hub is created in the namespace, click it and select Consumer Group in the left menu. Note that a dedicated Consumer Group for Devo needs to be created if the existing consumer groups are already in use.
Here you will see the Event Hub consumer groups. This will be used by the collector (or other applications) for reading data from the Event Hub. Write down the Consumer group name that you will use later in the configuration file.
Now, in the Event Hub Namespace, click on Shared access policies, search the default policy named RootManageSharedAccessKey and click it.
Copy and write down the primary (or secondary) connection string to be used later in the configuration file.
Setting up the Event Hubs
Now, search the Monitor service and click on it.
Click the Diagnostic Settings option in the left area.
A list of the deployed resources will be shown. Search for the resources that you want to monitor, select them, and click Add diagnostic setting.
Type a name for the rule and check the required category details (logs will be sent to the cloud.azure.eh.events table, and metrics will be sent to the cloud.azure.eh.metrics table).
Check Stream to an Event Hub, and select the corresponding Event hub namespace, Event hub name, and Event hub policy name.
Click Save to finish the process.
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).