Azure Event Hub collector

Purpose

An analyst wants to detect <adjective> behavior in <data source>.  Using the <name> Azure collector to send <type> to Devo, the analyst will find <outcome>.  As a result, the analyst will <verb> the <entity>, preventing  them from <tactic>.

Example tables

Authorize It

1. In Azure Portal, search for Entra ID.

   

2. Click App registrations in the left menu and click the app (or Service Principal) that you are going to use.

   

   Register the application

3. open Storage accounts.

2. On the Storage accounts page, select Create and name the account.

3. Click “Review + Create” then “Create”

4. After the storage account is created, select it from the list of storage accounts and click on Access keys in the left menu.
   

   

5. Copy the connection string.

Role assignment

Alternatively, users can grant the necessary permissions to the registered application to access the Event Hub without using the RootManageSharedAccessKey. Roles can be assigned in a variety of ways (e.g. inherited from the subscription group), but the following steps will show how to assign the necessary roles directly to the Storage Account.

Repeat steps 1-2 from the Connection String section to create the Storage Account.

1. In the Storage Account, click Access control (IAM) in the left menu, click + Add, and click Add Role Assignment.

   

   

2. Search for either the Storage Blob Data Contributor or Storage Blob Data Owner ??Storage Blob Data Reader?? role and select it and then click Next.

3. Click + Select members and search for the event hub application, select it, click Next.
   

   

4. Click Review + Assign.

5. Connection string

   Users can either obtain a connection string or use Role Assignments to allow the collector to access the Event Hub.

   1. In your Azure account, search for the Event Hubs service and click on it. 

   

   • Create an Event Hub resource per region (repeat the steps below for each region):

   • Click Add.

   

   • Fill the mandatory fields keeping in mind that the Event Hub must be in the same region as the resources that you are going to monitor
     To capture Blob or Data Lake, see How Event Hubs Capture is charged to select a tier. Otherwise, select the cheapest tier and one throughput unit. If you need more resources, they can be added later.

   

   • Select “Review+Create,” then “Create.”

   • The previous steps create an EventHub namespace; now go to Event Hubs, search the created one and click on it.

     

   • Now click on the + Event Hub button and create a new Event Hub

     

     Add a name.

   • One partition count is usually enough.

     Select the maximum retention time.

     

   • Once the Event Hub is created in the namespace, click it and select Consumer Group in the left menu. The $Default consumer group is fine. Devo recommends that the event hub only be accessed by Devo. If any other entity is accessing the event hub, then each entity must have its own consumer group and Microsoft must be paid more.

     

   • Now, in the Event Hub Namespace, click on Shared access policies, search the default policy named RootManageSharedAccessKey and click it.

   

   • Copy and write down the primary (or secondary) connection string to be used later in the configuration file.

Role assignment

Alternatively, users can grant the necessary permissions to the registered application to access the Event Hub without using the RootManageSharedAccessKey. Roles can be assigned in a variety of ways (e.g. inherited from the subscription group), but the following steps will show how to assign the necessary roles directly to the Event Hub Namespace.

Repeat all steps except the last one from the previous section to create the Event Hub.

1. In the Event Hub Namespace, click Access control (IAM) in the left menu, click + Add, and click Add Access Role Assignment.

   

2. Search for either the Azure Event Hubs Data Receiver role and select it and then click Next.

   

3. Click + Select members and search for the previously created App registration.

   1. Select the Application by clicking its name

   2. Click Select confusing label!

4. Click Review + Assign.

5. Create a shared access policy for sending data to the event hub.

   

6. Now, search for the Monitor service and click on it.

2. Click the Diagnostic Settings option in the left area.
   

3. Select a resource and Add diagnostic setting.

4. Add diagnostic setting

   

5. Name the diagnostic setting.

6. Enable metrics and logs. The options will vary.

7. Enable “Stream to an event hub.”

8. Select the namespace, hub, and policy you created.

   

9. Click Save to finish the process.

Run It

In the Cloud Collector App, create an Azure Collector instance using this parameters template, replacing the values enclosed in < >.

Secure It

Monitor It

Create an inactivity alert to detect interruptions of transfer of data from the source to the SQS queue using the query

from TABLE 
where toktains(hostchain,"collector-") 
select split(hostchain,"-",1) as collector_id

Set the inactivity alert to keep track of the collector_id.