Devo Source
- Juan Tomás Alonso Nieto (Deactivated)
- Former user (Deleted)
Description
This unit is a Source unit type.
This unit is an eternal query reader, meaning it begins from the time you start the Flow, and ends only when stopped. Once defined, it reads data from a specified LINQ query in your Devo domain. Output events are generated both to signal query operations and to deliver query results.
You can also define subqueries in the subquery configuration tab to apply specific offset and interval values to the main query.
An event is sent to the init port to signal the start of the query. The query specified in the properties is executed, along with the unit predicates. Although a query is usually executed just once and will terminate when there is no more data, you can set an eternal query, for example, if the unit is reconfigured or restarted. The last processed event date is remembered, in order to avoid repeating delivered events.
An event is sent through the data port for each result row.
The error port outputs an event every time there's an error running the query, retrieving new records, etc. The unit adds the field "action" to the output events with the value 'error', plus other standard error fields containing details about the error.
The stall port emits an event every time a query enters into stall mode. This is when the unit has not received new events for some time, it will continue running and will not end despite the apparent broken connection.Â
Finally, it is sent through the end port every time a query ends, be it after an error, or when there are no more results.Â
Configuration
After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:
Tab | Field | Description |
|---|---|---|
General | Name | Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed. |
Description | Enter a description detailing the scope of the unit. | |
Check DB configuration | Whether to check the query connection to the database before running it. This actually tries to connect to the given DB with provided parameters. | |
Query | Enter the LINQ query to be read. | |
Time column | Mandatory field. Specify the name of the column that contains the event timestamps in the queried table. By default, it shows eventdate. | |
Start time | You can specify a start time if you wish to test the query without input events. The unit will start sending data from that time included. | |
End time | You can specify an end time if you wish to test the query without input events. The unit will stop sending data from this time (excluded). | |
Time Zone | Enter the time zone to be used to schedule the unit. | |
Ensure all data | If this toggle is on, the unit will wait for all the query data to be present before sending it. It is enabled by default. | |
Temporal grouping saturation | Ask the query engine to saturate temporal grouping queries by adding an extra event for each absent period. Those extra events are built with the correct value for temporal grouping keys, with a null for any other non-temporal grouping key, and with whatever aggregations involved to return when empty. | |
End query when no more data | When turned on, the query is stopped when the time reaches the present time and no more data is available. The query ends on the first stall message received. | |
Wait on error | Time (in milliseconds) to wait before retrying after an error. | |
Force exploded mode | If this is set, the output event has a field named after each query column, containing the corresponding value, even if other output fields are already configured. The presence of 'valuesOn', 'pairsOn', or 'mapOn' deactivates exploded mode. If you want to force it, turn this on. | |
Fields on | You may add a field containing all the column names of your query to the events emitted by the output port data of the unit. To do it, specify a name for the column in this field. | |
Field list on | You may add a field containing all the column names of your query (following query order) to the events emitted by the data output port. To do it, specify a name for the column in this field. | |
Values on | If this is defined, the output event has a field with this name, containing a list of column values, in query order. | |
Pairs on | If this is defined, the output event has a field with this name, containing a list of name/value pairs (Map.Entry), in query order. | |
Map on | If this is defined, the output event has a field with this name, containing a map from column names to values. | |
Subquery configuration | External offset | Indicates the offset in milliseconds to apply to the outer query. Valid values range from 0 to 100 days (8640000000 milliseconds). |
Internal period | Indicates the offset in milliseconds to apply to the outer query. Valid values range from 0 to 100 days (8640000000 milliseconds). | |
Internal offset | Indicates the offset in milliseconds to apply to the outer query. Valid values range from 0 to 100 days (8640000000 milliseconds). |
Input ports
This unit does not have any input ports.
Output ports
Port | Description |
|---|---|
init | Outputs an event every time a new query is launched (on each input event). The events generated by this port have a column action that shows the value init. |
data | Outputs an event every time a new query record is available. The structure of the event depends on several configuration fields.
The presence of 'valuesOn', 'pairsOn', or 'mapOn' deactivates exploded mode. If you want to force it, set exploded mode on. |
error | Outputs an event every time there's an error, be it because the query could not be launched, or because there was some problem after that. The events generated by this port have a column action that shows the value error, plus two standard fields defining the error type (error and exception). |
end | Outputs an event every time a query launched ends successfully. The events generated by this port have a column action that shows the value end. |
stall | Emits an event every time a query that was launched enter in stall mode. Generated event has field 'eventdate' with current Timestamp. |
Example
In this example, we want to source data from a Devo data table containing web activity, and send details of country and city connections via email.
To do this, we will add a Devo Source unit with the following query:
from siem.logtrust.web.activityselect *
Then, we will add a Map unit to specify the fields to add, in this case the Java String, City and Country.
Link the data port of the Devo Source unit to the in port of the Map unit.
Finally, add an Email Sink unit to define the recipients. In the To recipients tab, add as many users as required.
Download this example
You can try this flow by downloading the following JSON file and uploading it to your domain using the Import option:
Â