Batch Detector

[ 1 Description ] [ 2 Configuration ] [ 3 Input ports ] [ 4 Output ports ] [ 5 Example ]

Description

This unit is a Processor unit.

A unit that signals the start and end of a batch of events, where a batch is considered to be a sequence of events that share a common value in a given field. This field is identified as the Input batch field value in the unit properties.

There are two types of input events for this unit:

Normal: input events that can produce both data and signal events on output.
Signal: input events that can only produce signals.

Both of them should contain the field configured in the Input batch field option of the unit's settings.

Normal events enter through the in port. If they belong to the same batch as the previous event, they will be forwarded through the data port. Otherwise, signals will be generated on start/end ports, and then the input event will be forwarded through the data port.

Configuration

After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:

Tab	Field	Description

Tab	Field	Description
General	Name	Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed.
	Description	Enter a description detailing the scope of the unit.
	Input batch field	This is where you configure which field will be used as the batch value, that is to say, the field that contains the value that all the events in a batch share.
	Output batch field	Name of the output field that will contain the relevant batch value. For start events, the batch that is starting; for end events, the batch that is ending; for data events, the current batch.
	Language	Specify the language you will use to write the expression in the expression fields, e.g. Javascript, Groovy, etc.
	Discard expression	Enter an expression to tell the unit when to discard an event or value, for example if there is a large difference between the current value and new value. In the expression, use "currentValue" and "newValue" variables respectively.
	Batch expression	Enter an expression for to obtain the batch value. If left empty, this field will only signal when the batch value changes from currentValue to newValue.

Input ports

Port	Description

Port	Description
in	"Normal" input events (will generate output signals if needed and then be forwarded through the data port).
signal	"Signal" input events (will generate output signals if needed, but won't be forwarded through the data port).

Output ports

Port	Description

Port	Description
start	Signals the start of a new batch. The output events are input events, enriched with batchIdField, containing the starting batch value.
data	Input events that are part of the current batch. The output events are input events, enriched with batchIdField, containing the starting batch value.
discarded	Input events that are discarded by the discard expression field. Output events are input events enriched with the field batchIdField, containing the starting batch value.
end	Signals when a batch ends. Events are clones of input events, enriched with batchIdField, containing the ending batch value.
error	Signals when an error occurred. Events are input events enriched with standard error fields.

Example

Imagine you have a Devo domain with users from two different countries and need to know each time users from one country or the other one are working in the domain.

You can use the Batch Detector unit to get notified each time the value in the country column of the siem.logtrust.web.activity table changes. To do it, add a Devo Source unit to indicate the source table that will send the events. Then, connect it to the Batch Detector unit through its in input port. Finally, link the start output port of the unit to an Email Sink unit to get notified each time a new batch of events starts.

In the Batch Detector unit properties, you must select the country column of the table, whose values will be used to define the different batches.

Download this example

You can try this flow by downloading this JSON and uploading it to your domain using the Import option: