Lookup - Flow unit

[ 1 Description ] [ 2 Configuration ] [ 3 Input ports ] [ 4 Output ports ] [ 5 Example ]

Description

This unit is a Processor unit type and is used to inspect and manage lookup table values.

Events are used both to signal transactions and to deliver results.

The first step is to load a lookup to the unit using the top five input ports. Lookup events are received via these ports as either signals to trigger a transaction, or as input events, depending on the input port.

When a query starts, a signal is sent to the begin port to start loading lookup data. This triggers a sequence to carry out various actions on the content (clear/update/delete) until reaching the end port, which triggers the input stream to finish. Lookup values are provided to the unit via the update port.

The in port is used to provide existing input events to be enriched with the lookup values provided in the update port. When the value specified in the lookup key column field of the existing lookup matches one of the columns of the input event, the values are enriched and output through the out port.

If there is no corresponding value, it is sent to the discarded port. If there are errors produced during the evaluation, these events are enriched with two additional fields, error and exception, and the event is output through the error port.

If you have set an expiration time in the properties, expired key-value pairs are sent through the expired port (see more about this below).

Configuration

After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:

Tab	Field	Description

Tab	Field	Description
General	Name	Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed.
	Description	Enter a description detailing the scope of the unit.
	Key field(s)	The name of input event field(s) containing key(s).
	Lookup key field(s)	The name of the lookup input event field(s) used to match with the key fields. These lookup events are provided via the update port.
	Lookup value field	The name of the event field to store in the lookup table as a value. This tells the unit into which key column to insert them.
	Expiration time	The number of milliseconds to wait before removing a key from the lookup table. If no key-value pair has been executed after this time has elapsed, the will be erased. Use 0 for no expiration. The notion of time is provided by the timestamp field.
	Lookup timestamp field	The name of the lookup input event field from the update and delete ports to use as the timestamp. Mandatory when the expiration time is > 0.
	Timestamp field	The name of the event field from the in port to use as the timestamp. Mandatory if the expiration time is > 0.
	Renew	Activate this option (when expiration time > 0) to update the key-value pair timestamp keys when used. This will update the expiration time of these keys so that they not considered as 'expired' even though they have been used.
	Output field name	The name of an output event field where the expression result will be inserted.
	Overwrite	Whether to overwrite an existing input event field if one with the same name already exists.

Input ports

Port	Description

Port	Description
begin	Stream of events to signal the start of a transaction.
clear	Stream of events used to clear the content of the lookup table.
update	Stream of events used to update values from the lookup table.
delete	Stream of events used to remove values from the lookup table.
end	Stream of events to signal the end of a transaction.
in	Events to be enriched with the table lookup values.

Output ports

Port	Description

Port	Description
out	Outputs events that enter through the in port, enriched with the values in the lookup table if the key value is found.
discarded	If the key value is not found in the in events, they come through this port.
expired	Expired key-value sets are sent through this port.
late	If a key value is late, i.e. enters after the timestamp, it leaves through this port.
error	Signals when an error occurred. Events are enriched with two additional fields, error and exception, and sent through this port.

Example

In this example, we want to use a lookup to enrich the main query with the data of a second query in order to define a subquery.

To do it, we will add a Generator unit that will run the queries every second. Then, we will define the main query and the subquery using two Full Query units. We will send the data of the main query through the in port of the Lookup unit and will use the data of the subquery to feed the lookup, using the top 5 input ports of the Lookup unit. Then, we will filter the output data using a Filter unit to get only the required fields and will finally send the data to a specific my.app table using a Devo Sink unit.

These are the configuration settings required for the Lookup unit:

Download this example

You can try this flow by downloading the following JSON file and uploading it to your domain using the Import option: