Devo Source

[ 1 Description ] [ 2 Configuration ] [ 3 Input ports ] [ 4 Output ports ] [ 5 Example ]

Description

This unit is a Source unit type.

This unit is an eternal query reader, meaning it begins from the time you start the Flow, and ends only when stopped. Once defined, it reads data from a specified LINQ query in your Devo domain. Output events are generated both to signal query operations and to deliver query results.

You can also define subqueries in the subquery configuration tab to apply specific offset and interval values to the main query.

An event is sent to the init port to signal the start of the query. The query specified in the properties is executed, along with the unit predicates. Although a query is usually executed just once and will terminate when there is no more data, you can set an eternal query, for example, if the unit is reconfigured or restarted. The last processed event date is remembered, in order to avoid repeating delivered events.

An event is sent through the data port for each result row.

The error port outputs an event every time there's an error running the query, retrieving new records, etc. The unit adds the field "action" to the output events with the value 'error', plus other standard error fields containing details about the error.

The stall port emits an event every time a query enters into stall mode. This is when the unit has not received new events for some time, it will continue running and will not end despite the apparent broken connection.

Finally, it is sent through the end port every time a query ends, be it after an error, or when there are no more results.

Configuration

After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:

Tab	Field	Description

Tab	Field	Description
General	Name	Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed.
	Description	Enter a description detailing the scope of the unit.
	Check DB configuration	Whether to check the query connection to the database before running it. This actually tries to connect to the given DB with provided parameters.
	Query	Enter the LINQ query to be read.
	Time column	Mandatory field. Specify the name of the column that contains the event timestamps in the queried table. By default, it shows eventdate.
	Start time	You can specify a start time if you wish to test the query without input events. The unit will start sending data from that time included.
	End time	You can specify an end time if you wish to test the query without input events. The unit will stop sending data from this time (excluded).
	Time Zone	Enter the time zone to be used to schedule the unit.
	Ensure all data	If this toggle is on, the unit will wait for all the query data to be present before sending it. It is enabled by default.
	Temporal grouping saturation	Ask the query engine to saturate temporal grouping queries by adding an extra event for each absent period. Those extra events are built with the correct value for temporal grouping keys, with a null for any other non-temporal grouping key, and with whatever aggregations involved to return when empty.
	End query when no more data	When turned on, the query is stopped when the time reaches the present time and no more data is available. The query ends on the first stall message received.
	Wait on error	Time (in milliseconds) to wait before retrying after an error.
	Force exploded mode	If this is set, the output event has a field named after each query column, containing the corresponding value, even if other output fields are already configured. The presence of 'valuesOn', 'pairsOn', or 'mapOn' deactivates exploded mode. If you want to force it, turn this on.
	Fields on	You may add a field containing all the column names of your query to the events emitted by the output port data of the unit. To do it, specify a name for the column in this field.
	Field list on	You may add a field containing all the column names of your query (following query order) to the events emitted by the data output port. To do it, specify a name for the column in this field.
	Values on	If this is defined, the output event has a field with this name, containing a list of column values, in query order.
	Pairs on	If this is defined, the output event has a field with this name, containing a list of name/value pairs (Map.Entry), in query order.
	Map on	If this is defined, the output event has a field with this name, containing a map from column names to values.
Subquery configuration	External offset	Indicates the offset in milliseconds to apply to the outer query. Valid values range from 0 to 100 days (8640000000 milliseconds).
	Internal period	Indicates the offset in milliseconds to apply to the outer query. Valid values range from 0 to 100 days (8640000000 milliseconds).
	Internal offset	Indicates the offset in milliseconds to apply to the outer query. Valid values range from 0 to 100 days (8640000000 milliseconds).

Input ports

This unit does not have any input ports.

Output ports

Port	Description

Port	Description
init	Outputs an event every time a new query is launched (on each input event). The events generated by this port have a column action that shows the value init.
data	Outputs an event every time a new query record is available. The structure of the event depends on several configuration fields. In exploded mode (default), each query column generates a field in the output event with the corresponding name and value. If you set: 'fieldsOn': A set containing all field names is added with the given name. 'fieldListOn': A list containing all field names, following query order, is added with the given name. 'valuesOn': A list containing all field values, following query order, is added with the given name. 'pairsOn': A list containing field name/value pairs, following query order, is added with the given name. 'mapOn': A map from field names to values is added with the given name. The presence of 'valuesOn', 'pairsOn', or 'mapOn' deactivates exploded mode. If you want to force it, set exploded mode on.
error	Outputs an event every time there's an error, be it because the query could not be launched, or because there was some problem after that. The events generated by this port have a column action that shows the value error, plus two standard fields defining the error type (error and exception).
end	Outputs an event every time a query launched ends successfully. The events generated by this port have a column action that shows the value end.
stall	Emits an event every time a query that was launched enter in stall mode. Generated event has field 'eventdate' with current Timestamp.

Example

In this example, we want to source data from a Devo data table containing web activity, and send details of country and city connections via email.

To do this, we will add a Devo Source unit with the following query:

from siem.logtrust.web.activity
select *

Then, we will add a Map unit to specify the fields to add, in this case the Java String, City and Country.

Link the data port of the Devo Source unit to the in port of the Map unit.

Finally, add an Email Sink unit to define the recipients. In the To recipients tab, add as many users as required.

Download this example

You can try this flow by downloading the following JSON file and uploading it to your domain using the Import option: