Document toolboxDocument toolbox

Devo Source

Description

This unit is a Source unit type.

This unit is an eternal query reader, meaning it begins from the time you start the Flow, and ends only when stopped. Once defined, it reads data from a specified LINQ query in your Devo domain. Output events are generated both to signal query operations and to deliver query results.

You can also define subqueries in the subquery configuration tab to apply specific offset and interval values to the main query.

An event is sent to the init port to signal the start of the query. The query specified in the properties is executed, along with the unit predicates. Although a query is usually executed just once and will terminate when there is no more data, you can set an eternal query, for example, if the unit is reconfigured or restarted. The last processed event date is remembered, in order to avoid repeating delivered events.

An event is sent through the data port for each result row.

The error port outputs an event every time there's an error running the query, retrieving new records, etc. The unit adds the field "action" to the output events with the value 'error', plus other standard error fields containing details about the error.

The stall port emits an event every time a query enters into stall mode. This is when the unit has not received new events for some time, it will continue running and will not end despite the apparent broken connection. 

Finally, it is sent through the end port every time a query ends, be it after an error, or when there are no more results. 

Configuration

After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:

Tab

Field

Description

Tab

Field

Description

General

Name

Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed.

Description

Enter a description detailing the scope of the unit.

Check DB configuration

Whether to check the query connection to the database before running it. This actually tries to connect to the given DB with provided parameters.

Query

Enter the LINQ query to be read.

Time column

Mandatory field. Specify the name of the column that contains the event timestamps in the queried table. By default, it shows eventdate.

Start time

You can specify a start time if you wish to test the query without input events. The unit will start sending data from that time included.

End time

You can specify an end time if you wish to test the query without input events. The unit will stop sending data from this time (excluded).

Time Zone

Enter the time zone to be used to schedule the unit.

Ensure all data

If this toggle is on, the unit will wait for all the query data to be present before sending it. It is enabled by default.

Temporal grouping saturation

Ask the query engine to saturate temporal grouping queries by adding an extra event for each absent period. Those extra events are built with the correct value for temporal grouping keys, with a null for any other non-temporal grouping key, and with whatever aggregations involved to return when empty.

End query when no more data

When turned on, the query is stopped when the time reaches the present time and no more data is available. The query ends on the first stall message received.

Wait on error

Time (in milliseconds) to wait before retrying after an error.

Force exploded mode

If this is set, the output event has a field named after each query column, containing the corresponding value, even if other output fields are already configured.

The presence of 'valuesOn', 'pairsOn', or 'mapOn' deactivates exploded mode. If you want to force it, turn this on.

Fields on

You may add a field containing all the column names of your query to the events emitted by the output port data of the unit. To do it, specify a name for the column in this field.

Field list on

You may add a field containing all the column names of your query (following query order) to the events emitted by the data output port. To do it, specify a name for the column in this field.

Values on

If this is defined, the output event has a field with this name, containing a list of column values, in query order.

Pairs on

If this is defined, the output event has a field with this name, containing a list of name/value pairs (Map.Entry), in query order.

Map on

If this is defined, the output event has a field with this name, containing a map from column names to values.

Subquery configuration

External offset

Indicates the offset in milliseconds to apply to the outer query.

Valid values range from 0 to 100 days (8640000000 milliseconds).

Internal period

Indicates the offset in milliseconds to apply to the outer query.

Valid values range from 0 to 100 days (8640000000 milliseconds).

Internal offset

Indicates the offset in milliseconds to apply to the outer query.

Valid values range from 0 to 100 days (8640000000 milliseconds).

Input ports

This unit does not have any input ports.

Output ports

Port

Description

Port

Description

init 

Outputs an event every time a new query is launched (on each input event). The events generated by this port have a column action that shows the value init.

data

Outputs an event every time a new query record is available. The structure of the event depends on several configuration fields.

  • In exploded mode (default), each query column generates a field in the output event with the corresponding name and value.

  • If you set:
                      'fieldsOn': A set containing all field names is added with the given name.
                      'fieldListOn': A list containing all field names, following query order, is added with the given name.
                      'valuesOn': A list containing all field values, following query order, is added with the given name.
                      'pairsOn': A list containing field name/value pairs, following query order, is added with the given name.
                      'mapOn': A map from field names to values is added with the given name.

The presence of 'valuesOn', 'pairsOn', or 'mapOn' deactivates exploded mode. If you want to force it, set exploded mode on.

error

Outputs an event every time there's an error, be it because the query could not be launched, or because there was some problem after that. The events generated by this port have a column action that shows the value error, plus two standard fields defining the error type (error and exception).

end

Outputs an event every time a query launched ends successfully. The events generated by this port have a column action that shows the value end.

stall

Emits an event every time a query that was launched enter in stall mode. Generated event has field 'eventdate' with current Timestamp.

Example

In this example, we want to source data from a Devo data table containing web activity, and send details of country and city connections via email.

To do this, we will add a Devo Source unit with the following query:

from siem.logtrust.web.activity
select *

Then, we will add a Map unit to specify the fields to add, in this case the Java String, City and Country.

Link the data port of the Devo Source unit to the in port of the Map unit.

Finally, add an Email Sink unit to define the recipients. In the To recipients tab, add as many users as required.

Download this example

You can try this flow by downloading the following JSON file and uploading it to your domain using the Import option:

Â