Devo Managed Query

[ 1 Description ] [ 2 Configuration ] [ 3 Input ports ] [ 4 Output ports ] [ 5 Example ]

Description

This unit is a Source unit and is a variant of the DevoSource unit.

The Devo Managed Query unit can be configured to optimize the management of eternal queries and data stalls. This means that when the unit has not received new events for some time, it will continue running and will not end despite the apparent broken connection.

This unit has 2 input ports:

do: Input events entering through this port will launch a new execution of the query, overriding the query that is already running.
stop: The reception of an event through this port stops the current query.

An event enters through the do port. Then, an event is sent to the init port to signal the start of the query. The query specified in the properties is executed. An event is sent through the data port for each result row.

The error port outputs an event every time there's an error running the query, retrieving new records, etc. The unit adds the field "action" to the output events with the value 'error', plus other standard error fields containing details about the error.

Finally, it is sent through the end port every time a query ends, be it after an error, or when there are no more results. These events are the same as the input events, without any modification.

Alternatively, if an event enters through the stop port, the query ends.

Configuration

After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:

Tab	Field		Description

Tab	Field		Description
General	Name		Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed.
	Description		Enter a description detailing the scope of the unit.
	Check DB configuration		Whether to check the connection to the DB when the unit is configured. This actually tries to connect to the given DB with provided parameters.
	Poll wait time		Time (in milliseconds) to wait between query iterations when the unit is in a data stall in order to avoid closing the query.
	Max loops		The maximum number of iterations of the query.
	Query		The Devo query to be performed, in LINQ format. This query can include parameters (defined in the 'parameters' field), that will be filled with the corresponding input event field.
	Start time		You can specify a start time if you wish to test the query without input events. The unit will start sending data from that time included.
	End time		You can specify an end time if you wish to test the query without input events. The unit will stop sending data from this time (excluded).
	Parameters		A list of field names to be used as parameters in the query, evaluated in order. Click the plus icon to add. You can collapse and expand the parameters using the icons, reorder using the arrows, or delete using the X icon.
	Fields on		When defined, this is the name of the output event field with a list of column names.
	Map on		When defined, this is the name of the output event field if you wish to map from column names to values.
	Ensure all data		If this is set, the query will wait to obtain all data before beginning.
Columns	Event field		The name of an input event field containing a list of (name, type) pairs.
	Fields	Click the plus icon to add name-type pairs. You can collapse and expand the pairs using the icons, reorder using the arrows, or delete using the X icon. You can also reorder the fields using drag and drop.
		Name	The name of an output event field where the query result will be inserted.
		Type	The expected type for the query result (java class qualified name).

Input ports

Port	Description

Port	Description
do	Input events entering through this port will launch a new execution of the query, overriding the query that is already running.
stop	The reception of an event through this port stops the current query.

Output ports

Port	Description

Port	Description
init	Outputs an event every time a new query is launched (on each input event). These events are the same as the input events, without any modification.
data	Outputs an event for each query result row. The events generated are the input events extended with fields including the query results. Further required columns can be specified in the unit configuration options.
error	Outputs an event every time there's an error running the query, retrieving new records, etc. The generated event is an input event enriched with standard error messages.
end	Outputs an event every time a query ends, be it after an error, or when there are no more results. These events are the same as the input events, without any modification.
stall	Outputs an event every time a query that was launched enters into stall mode. The generated event has the field 'eventdate' with the current Timestamp.

Example

In this example, we want to source data on usernames and timestamps using a dynamic query, and send the results to a my.app table.

To do this, we add a Tick unit. In the Fields tab of the properties, add the username fields to send, changing user to your username.

Next, we will add a Devo Managed Query unit to search for a dynamic username and specific time window using the following query:

from siem.logtrust.web.activity 
select * where username = ? 
and now() - 3h < eventdate < now()

Enter the username parameter in the Parameters field below.

Link the out port of the Tick unit to the do port of the Devo Managed Query unit.

Finally, add a Devo Sink unit to define the my.app destination table. Link the data out port of the Devo Managed Query to the in port and click the Tick to begin.

After saving and starting the Flow, you must click the red button of the Tick unit to begin sending events.

Download this example

You can try this flow by downloading the following JSON file and uploading it to your domain using the Import option: