Scaling the Event Hub collector
- Laszlo Frazer
Analytics
The default Event Hub collector configuration almost always runs fast enough. Event hub collectors can be scaled up by using multiple pods. Virtual Machine Metrics collectors, which are also Azure collectors, must not use multiple pods.
Before increasing the number of pods for an event hub collector:
Verify the event hub is scaled correctly in Azure.
Determine the number of partitions enabled in Azure.
Determine the collector delay using this Devo query.
from cloud.azure
select eventdate-parsedate(str(jqeval(jqcompile(".\"@devo_event_enqueued_time\""), jsonparse(rawMessage))),"YYYY-MM-DD[T]HH:mm:ss.SSSSSS[Z]","utc") as collector_delay,
parsedate(str(jqeval(jqcompile(".\"@devo_event_enqueued_time\""), jsonparse(rawMessage))),"YYYY-MM-DD[T]HH:mm:ss.SSSSSS[Z]","utc")-parsedate(timestamp,"YYYY-MM-DD[T]HH:mm:ss.SSSSSSS[Z]","utc") as eventhub_delay,
collector_delay+eventhub_delay>1h as problem_detectedIf the collector delay is more than one hour and is not decreasing, then:
In the collector parameters, add
"client_thread_limit": 1,to the configuration of the queue.
Increase the number of pods to be equal to the number of partitions.
If you are unable to increase the target pods due to the message
Only target pods between 0 and … are permitted
then please visit the support site for assistance.