Memory

[ 1 Description ] [ 2 Configuration ] [ 3 Input ports ] [ 4 Output ports ] [ 5 Example ]

Description

This unit is a Processor unit type.

This unit can be used to store key(s)/value(s) in its memory. You can use several input fields as a multi key, and one input field as a value. This is a very flexible unit that can be used for many different purposes. You can configure an unlimited number of input/output ports with as many functionalities as required.

Data is stored in a multivalued table, 'mapping' each key-values list pair. Each mapping has a timestamp of the moment it was last used. The same applies for each individual value.

The unit is triggered upon receiving input events, or when a bound is fulfilled.

Input event handling starts with an action execution phase:

An input event comes through one of the configured input ports. Depending on the settings for that port, input events need to contain key(s) and value, just key(s), or nothing.

A sequence of actions is executed. Actions will either be successfully completed (e.g. delete existing key(s)) or will fail (e.g. attempt to delete key(s) that do not exist). Even when an action fails, the following actions are still executed, as actions are considered individually.

Actions to be performed on an input event are deconstructed into 'segments', each with a success/failure branch. This recursive structure defines a tree. With any input event, a path on this tree is traversed until a leave node is reached, executing actions it comes across along the way. For more information, see Decision Tree Learning.

Successful actions will be sent to the 'Success' branch.

Failed actions take the 'Failure' branch.

Then, successful events are generated. The path is traversed from end to beginning and, at each segment, output events for each action are sent to the specified out port.

Failed events are generated in the same way, this time with a 'reason' event sent to the specified destinations. A reason event is the input event enriched to indicate the first failed action on the segment.
Finally, the bounds are processed. Fulfilling a bound will trigger specified actions, and events are sent to given destinations.

The process is the same when triggered by a timeout bound.

Configuration

After dragging this unit into the Flow canvas, double-click it to access its configuration options. The following table describes the configuration options of this unit:

Tab	Field		Description

Tab	Field		Description
General	Name		Enter a name for the unit. It must start with a letter, and cannot contain spaces. Only letters, numbers, and underscores are allowed.
	Description		Enter a description detailing the scope of the unit.
	Key field(s)		Enter the name of the input event fields containing key(s). The unit can be configured to either use a global port for all events or to create a new port for each key.
	Value field		Enter the name of the input event field containing the value. The unit can be configured to either use a global port for all events or to create a new port for each value.
	Allow duplicates		Whether to repeat the same (key(s), value) pair. The functionality of some actions depend on this.
	Allow empty collections		Whether empty collections should emit a result or not (false by default). This only applies when a value is set in the 'Collect' setting of the output port.
	Replace on import		If you activate this option, receiving an event from a port with the 'import' action will replace the data received. Otherwise, it will append the information to the internal store.
Input ports	Input ports	Click the plus icon to add as many additional input ports as needed. You can collapse and expand the ports using the icon, reorder using the arrows, or delete using the X icon. You can also reorder the fields using drag and drop.
		Name	Enter a name for the input event port containing the key(s).
		Key field(s)	Enter the name of the input event field containing key(s). This overrides the unit-level Key fields setting for this particular input port.
		Value field	Enter the name of the input event field containing the value. This overrides the unit-level Value field setting for this particular input port.
		Key index(es)	Select the position of the key field(s) with respect to the full key used in the stored action. The numbering is zero-based (0 is the first position).
		Actions	A list of actions to execute. Describe actions to be taken using clear, concise patterns to facilitate the creation of custom functionality. For example, if you want to save an entry, you have to specify both 'store' and 'touch'. Actions that require key(s) will fail if the key is not found. Some functionality will depend on whether you have set Allow duplicates to on or off. Possible actions: Pass: Requires key(s) & value on input event. Passes content as output event. Store: Requires key(s) & value on input event. Associates value with key(s). Saves creation timestamp. Does not generate output event. Fails if key(s) & value already exist. StoreUnique: Requires key(s) & value on input event. Associates value with key(s). Saves creation timestamp. Does not generate output event. Fails if key(s) & value already exist. RetrieveAll: Requires key(s) & value on input event. Extracts all values associated to key(s). Fails if no value exists. Retrieve1: Requires key(s) & value on input event. Extracts one value (undefined) associated to key(s). Fails if no value exists. RetrieveKey: Requires key(s) on input event. Extracts all values associated to key(s). Fails if no value exists. RetrieveKeyIncomplete: Requires key(s) on input event and index(es). Extracts all values associated to an incomplete key(s). Fails if no value exists or indexes are wrong. Dump: Requires nothing on input event. Extracts all values. History: Same as Dump, but values are sorted by arrival order. Delete, Remove: Requires key(s) & value on input event. Deletes (key(s), value) association. Fails if association not found. DeleteAll, RemoveAll: Requires key(s) & value on input event. Deletes all associations. Fails if no association is found. Delete1, Remove1: Requires key(s) on input event. Deletes a value (undefined) associated to key(s). Fails if no value found. DeleteKey, RemoveKey: Requires key(s) on input event. Deletes all values associated to key(s). Fails if no value found. Clean, Clear: Requires nothing in input event. Empties memory. Touch: Requires key(s) & value on input event. Updates association (not mapping) timestamp. If several associations found, update only one (undefined). Fails if association not found. TouchAll: Requires key(s) & value on input event. Updates association (not mapping) timestamp. If several associations found, updates all. Fails if association not found. Touch1: Requires key(s) on input event. Searches for an association with that key(s). Updates association (not mapping) timestamp. If several associations found, update only one (undefined). Fails if association not found. TouchKeyValues: Requires key(s) on input event. Searches for associations with that key(s). Updates associations' (not mapping) timestamp. If several associations found, update all. Fails if association not found. TouchKey: Requires key(s) on input event. Searches for mapping with that key(s). Updates mapping (not associations) timestamp. Fails if mapping not found.
		Destinations	Select the output port to which events will be sent.
		Success	The branch to be taken when all actions in this segment are successful. You can add as many success branches and actions as needed.
		Failure	Branch to be taken when an action in this segment fails. All actions in the segment are executed, even if some of them fail. You can add as many failure branches and actions as needed.
Output ports	Output ports	Click the plus icon to add a list of output ports. You can collapse and expand the ports using the icon, reorder using the arrows, or delete using the X icon. You can also reorder the fields using drag and drop.
		Name	Enter a name for the output port.
		Collect	Type of grouping applied to output events coming through this port. Possible types: Nothing: Individual events for each association are generated. Counter is always one. Date is the arrival date. Samekeyvalue: One event for each different association is generated. Counter is the number of equal associations. Date is the oldest date found. Samekey: One event for each key(s) is generated. All: One event with all (key(s), value) pairs, grouped.
		Collect as list	Whether the collection type is Collection (true) or Array (false).
		Key(s) field	Enter a name for the output event field containing key(s).
		Value(s) field	Enter a name for the output event field containing value(s).
		Count(s) field	Enter a name for the output event field containing counter(s). This is 1 when there's no grouping, and the number of grouped items otherwise.
		Percent(s) field	Enter a name for the output event field containing percent(s). The percent is the value of the counter with respect to the total elements in memory (a number between 0 and 1).
		Report field	The name of the output event field containing the report. The report is a map of two values: Absolute: a. the total size of the memory. b. the number of different values for each key field. c. the size of the result set of the query. perKey: An array with detailed information for each key present in the result set. Each element of the array contains the following information: valueCount: number of entries in the result set (fixing all levels in key). keyXValueCount: number of entries fixing the current level and previous ones. keyXTotalValueCount: number of entries fixing only the current level. For the first key, equal to keyXValueCount. keyXDistinctKeyCount: number of options at position X fixing previous levels. keyXDistinctKeyValueCount: number of entries fixing previous levels. keyXValuePercent: percentage of entries, fixing current and previous levels, relative to those fixing only previous levels. This is, keyXValueCount/key(X-1)ValueCount. For the first level, relative to full memory. keyXTotalValuePercent: percentage of entries, fixing only current level, relative to full memory. This is, keyXTotalValueCount/size. For the first level, equal to keyXValuePercent.
		Time field	Enter a name for the output event field containing the timestamp. When groupings are present, the timestamp will be the oldest found.
		Error field	Enter a name for the output event field containing the error. Error is the name of the first action that failed in a segment, with this port configured as one of the destinations of its 'reason'.
Bounds	Bounds	Bounds are a way of establishing limits that, when reached, will generate some reaction. Currently, you can set size (max number of items) and time (ttl, timeout) limits. Click the plus icon to add bounds. You can collapse and expand the bounds using the icon, reorder using the arrows, or delete using the X icon. You can also reorder the fields using drag and drop.
		Actions	A list of actions to execute. For time bounds, the possible actions are renew (default) or remove. This setting does not apply for size bounds, as the only possible action is remove.
		Destinations	Select the output port to which events will be sent.
		Seconds	Amount of time to wait (in seconds) before purging events from the memory.
		Size	The maximum number of elements to store in the memory.
		Purge size	The number of elements to purge from the memory when it is full (1 by default). This only applies when byValue is on.
		Bound per Key	Whether previous bound settings refer to individual keys or whole memory. This setting only applies to size bounds.
		By value	Whether previous bound settings refer to individual values or whole mappings.

Input ports

Add as many input ports as you need. Names must be unique within the unit.

Output ports

Add as many input ports as you need. Names must be unique within the unit.

Example

In this example, we want to use a Memory unit that receives various machines via different input ports. When the machine exists, the data will be enriched with up to date values. When it does not exist, the memory unit is enriched with new values. This way, the memory unit stores the latest machine data before outputting it for further analysis in a my.app table.

For this, we will add two Tick units to specify the fields to send to the Memory unit. The first Tick sends machine data as a string and ratio data as an integer:

The second sends different machine data:

Next, add the Memory unit to the canvas. In the properties, specify the name of the output event in the Value field, in this case Value, and turn on Replace on import.

Next, in the Input ports tab, we will specify the input ports for both Ticks using key-value pairs.

The first port inputs new values, deleting existing values if found. The key field corresponds to the field specified in the Tick properties, in this case Machine. The value field is the second field configured, in this case the integer someRatio.

As we wish to store, or update when present, the new value associated to the incoming key, we first try to delete the stored key-value using DeleteKey, RemoveKey in the Actions field.

After that, both if it succeeded or not, we store the new pair key-value. For this, enter store as the Success and Failure actions.

The second input port serves to enrich the destination table via the out port specified.

Add the following properties:

Link each Tick unit to its assigned input port.

Finally, add a Devo Sink unit and specify the my.app table to be enriched with data.

After saving and starting the Flow, you must click the red button of each Tick unit to begin sending events.

Download this example

You can try this flow by downloading the following JSON file and uploading it to your domain using the Import option: