Devo Managed Query
- Juan Tomás Alonso Nieto (Deactivated)
Description
This unit is a Source unit and is a variant of the DevoSource unit.
The Devo Managed Query unit can be configured to optimize the management of eternal queries and data stalls. This means that when the unit has not received new events for some time, it will continue running and will not end despite the apparent broken connection.
This unit has 2 input ports:
do: Input events entering through this port will launch a new execution of the query, overriding the query that is already running.
stop: The reception of an event through this port stops the current query.
An event enters through the do port. Then, an event is sent to the init port to signal the start of the query. The query specified in the properties is executed. An event is sent through the data port for each result row.
The error port outputs an event every time there's an error running the query, retrieving new records, etc. The unit adds the field "action" to the output events with the value 'error', plus other standard error fields containing details about the error.
Finally, it is sent through the end port every time a query ends, be it after an error, or when there are no more results. These events are the same as the input events, without any modification.
Alternatively, if an event enters through the stop port, the query ends.
Configuration
After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:
Tab | Field | Description | |
|---|---|---|---|
General | Name | Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed. | |
Description | Enter a description detailing the scope of the unit. | ||
Check DB configuration | Whether to check the connection to the DB when the unit is configured. This actually tries to connect to the given DB with provided parameters. | ||
Poll wait time | Time (in milliseconds) to wait between query iterations when the unit is in a data stall in order to avoid closing the query. | ||
Max loops | The maximum number of iterations of the query. | ||
Query | The Devo query to be performed, in LINQ format. This query can include parameters (defined in the 'parameters' field), that will be filled with the corresponding input event field. | ||
Start time | You can specify a start time if you wish to test the query without input events. The unit will start sending data from that time included. | ||
End time | You can specify an end time if you wish to test the query without input events. The unit will stop sending data from this time (excluded). | ||
Parameters | A list of field names to be used as parameters in the query, evaluated in order. Click the plus icon to add. You can collapse and expand the parameters using the icon, reorder using the arrows, or delete using the X icon. | ||
Fields on | When defined, this is the name of the output event field with a list of column names. | ||
Map on | When defined, this is the name of the output event field if you wish to map from column names to values. | ||
Ensure all data | If this is set, the query will wait to obtain all data before beginning. | ||
Columns | Event field | The name of an input event field containing a list of (name, type) pairs. | |
Fields | Click the plus icon to add name-type pairs. You can collapse and expand the pairs using the icon, reorder using the arrows, or delete using the X icon. You can also reorder the fields using drag and drop. | ||
Name | The name of an output event field where the query result will be inserted. | ||
Type | The expected type for the query result (java class qualified name). | ||
Input ports
Port | Description |
|---|---|
do | Input events entering through this port will launch a new execution of the query, overriding the query that is already running. |
stop | The reception of an event through this port stops the current query. |
Output ports
Port | Description |
|---|---|
init | Outputs an event every time a new query is launched (on each input event). These events are the same as the input events, without any modification. |
data | Outputs an event for each query result row. The events generated are the input events extended with fields including the query results. Further required columns can be specified in the unit configuration options. |
error | Outputs an event every time there's an error running the query, retrieving new records, etc. The generated event is an input event enriched with standard error messages. |
end | Outputs an event every time a query ends, be it after an error, or when there are no more results. These events are the same as the input events, without any modification. |
stall | Outputs an event every time a query that was launched enters into stall mode. The generated event has the field 'eventdate' with the current Timestamp. |
Example
In this example, we want to source data on usernames and timestamps using a dynamic query, and send the results to a my.app table.
To do this, we add a Tick unit. In the Fields tab of the properties, add the username fields to send, changing user to your username.
Next, we will add a Devo Managed Query unit to search for a dynamic username and specific time window using the following query:
from siem.logtrust.web.activity
select * where username = ?
and now() - 3h < eventdate < now()Enter the username parameter in the Parameters field below.
Link the out port of the Tick unit to the do port of the Devo Managed Query unit.
Finally, add a Devo Sink unit to define the my.app destination table. Link the data out port of the Devo Managed Query to the in port and click the Tick to begin.
After saving and starting the Flow, you must click the red button of the Tick unit to begin sending events.
Download this example
You can try this flow by downloading the following JSON file and uploading it to your domain using the Import option:
