/
Scaling the Event Hub collector
Document toolboxDocument toolbox

Scaling the Event Hub collector

Owned by Laszlo Frazer
Last updated: Feb 26, 2025 by Virginia Camuñas Núñez
1 min read

Scope

The default Event Hub collector configuration almost always runs fast enough. Event hub collectors can be scaled up by using multiple pods. Virtual Machine Metrics collectors, which are also Azure collectors, must not use multiple pods.

Before increasing the number of pods for an event hub collector:

  • Verify the event hub is scaled correctly in Azure.

  • Determine the number of partitions enabled in Azure.

  • Determine the collector delay using this Devo query.

from cloud.azure select eventdate-parsedate(str(jqeval(jqcompile(".\"@devo_event_enqueued_time\""), jsonparse(rawMessage))),"YYYY-MM-DD[T]HH:mm:ss.SSSSSS[Z]","utc") as collector_delay, parsedate(str(jqeval(jqcompile(".\"@devo_event_enqueued_time\""), jsonparse(rawMessage))),"YYYY-MM-DD[T]HH:mm:ss.SSSSSS[Z]","utc")-parsedate(timestamp,"YYYY-MM-DD[T]HH:mm:ss.SSSSSSS[Z]","utc") as eventhub_delay, collector_delay+eventhub_delay>1h as problem_detected

If the collector delay is more than one hour and is not decreasing, then using the Cloud Collector App:

  1. In the collector parameters, add

"client_thread_limit": 1,

to the configuration of the queue.

  1. Increase the number of pods to be equal to the number of partitions.

  2. If you are unable to increase the target pods due to the message

Only target pods between 0 and … are permitted

image-20250128-161215.png

then please visit the support site for assistance.

 

Related content