Versions Compared

Old Version 2

changes.mady.by.user Sarah Bigault

Saved on

compared with

New Version Current

changes.mady.by.user Borja Moro Moreno

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents
minLevel21
maxLevel72
typeflat

Description

The flow shown below is an example of data injection. After having transformed our data passing through an external web service (RandomUser.me) and several components which act as data formatting, the data will be written into and read from a Devo table. Then, a second sub-flow will get the data from the injected table, configure it, and send the results via Slack.

...

Flow configuration

As you can see in the picture above, this flow is composed by 2 different sets of units: a main flow that gets the data, configures it, and injects it into a Devo table, and a sub-flow that will read the data and send it to a specified Slack channel.

Anchor
flowconfig
flowconfig

The following table describes the units needed to create the main flow, as well as how to configure and link them.

Unit type and description

Configuration

Tick

Image Removed
Image Added

This unit will send an event each time we click it, which will call an external web service added in the next unit.

Drag a Tick unit to the canvas and open its configuration options by double-clicking it.

In the General tab, simply add a name to the unit (in this example, we called it callService).

We don't need to add anything in the Fields tab since we only need to call a request to the external web service added in the following unit upon demand, that is to say, when clicking this unit.

Click Apply to save the configuration.

HttpCall

Image Removed
Image Added

This unit will call a request to the RandomUser API each time we click the Tick unit added. A list of random users will be added to the response column that will be added to the events.

Drag a HttpCall unit to the canvas and link the output port of the Tick unit to the input port of this unit.

Image Removed
Image Added

Then, open the configuration options of the unit by double-clicking it and enter the following values in the General tab:

Image Removed
Image Added

Click Apply to save the configuration.

JsonParser

Image Removed
Image Added

This unit will parse the response string generated by the RandomUser API and transform it into a JSON object.

Drag a JsonParser unit to the canvas and link the output port out of the HttpCall unit to the input port of this unit.

Image Removed
Image Added

Then, open the configuration options of the unit by double-clicking it and enter the following values:

  • Name - jsonParser

  • In field - Choose the response field generated by the HttpCall unit.

  • Out field - Type the name of the JSON field which will be created. In this example, we will call it data.

Image Removed
Image Added

Click Apply to save the configuration.

Map

Image Removed
Image Added

This unit will extract the required information from the received JSON objects.

Drag a Map unit to the canvas and link the output port out of the JsonParser unit to the input port of this unit.

Image Removed
Image Added

Then, open the configuration options of the unit by double-clicking it. In the General tab, simply add a name to the unit (in this example, we called it getFields).

Now go to the Fields to add tab. Click the + icon  in the fields section to add 8 new columns to the events sent:

Column 1

This column extracts all the info from the JSON object. Fro there, we will start extracting the required parts.

  • Field name - all

  • Type - Map

  • Expression - This is the expression that defines the new column:

Code Block
data.get("results").get(0)
Image Removed
Image Added

Column 2

This column extracts the name from the JSON object.

  • Field name - name

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all.get("name").get("first")
Image Removed
Image Added

Column 3

This column extracts the surname from the JSON object.

  • Field name - surname

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all.get("name").get("last")
Image Removed
Image Added

Column 4

This column extracts the gender from the JSON object.

  • Field name - gender

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all.get("gender")
Image Removed
Image Added

Column 5

This column extracts the email from the JSON object.

  • Field name - email

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all.get("email")
Image Removed
Image Added

Column 6

This column extracts the location from the JSON object.

  • Field name - location

  • Type - java.lang.Object

  • Expression - This is the expression that defines the new column:

Code Block
all.get("location")
Image Removed
Image Added

Column 7

This column extracts the city from the JSON object.

  • Field name - city

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
location.get("city")
Image Removed
Image Added

Column 8

This column extracts the country from the JSON object.

  • Field name - country

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
location.get("country")
Image Removed
Image Added

Click Apply to save the configuration.

Map

Image Removed
Image Added

This unit will delete the fields that we don't need from the received events.

Drag another 

Filter 

Map unit to the canvas and link the output port out of the previously added 

Filter

Map unit to the input port of this unit.

Image Removed
Image Added

Then, open the configuration options of the unit by double-clicking it. In the General tab, simply add a name to the unit (in this example, we called it removeGenericFields).

Now go to the Fields to remove tab. Click the Fields field to expand a dropdown menu that contains all the columns in your events. In this case, we will select these ones in order to remove them: response, data, all and location.

Image Removed
Image Added

Click Apply to save the configuration.

DevoSink

Image Removed
Image Added

This unit will inject the received data to a Devo my.app table.

Drag a DevoSink unit to the canvas and link the output port out of the second 

Filter

Map unit to the input port of this unit.

Image Removed
Image Added

Then, open the configuration options of the unit by double-clicking. Enter a Name for the unit (in this example, we called it sendToDevo) and the Table name where you want to send the data. 

Note

Note that the table you choose must be a my.app table. This works as Devo injections, which can only be done in my.app tables. Learn more in Inject data to a new table.

Image Removed
Image Added

Click Apply to save the configuration.

Now that we have created a Flow to send data to the specified my.app table, we will create another one that will read the data in that table, format it and send it to a specified Slack channel.

Uni type and description

Configuration

DevoSource

Image Removed
Image Added

This unit will read the data from the my.app table where we injected the events.

Drag a DevoSource unit to the canvas and open its configuration options by double-clicking it.

Enter a Name for the unit (in this example, readFromDevo)and check off for the Check DB configuration field. In the Query field, enter the query from which you want to get data. In this example, we will use the following:

Code Block
from my.app.test
where format = "simpleetl"
select *

Also, you can optionally choose a Start time for the query.

Image Removed
Image Added

Click Apply to save the configuration.

Map

Image Removed
Image Added

We will use this unit to get the information we want to send through Slack and format it as required.

Drag a Map unit to the canvas and link the output port data of the DevoSource unit to the input port of this unit.

Image Removed

Then, open the configuration options of the unit by double-clicking it. In the General tab, simply add a name to the unit (in this example, we called it messageToEvent).

Now go to the Fields to add tab. Click the + icon in the fields section to add 6 new columns to the events sent, including the required format we will use to send the messages through Slack:

Column 1

  • Field name - all

  • Type - java.lang.String[]

  • Expression - This is the expression that defines the new column:

Code Block
message.split("\\^\\^#\\|")
Image Removed
Image Added

Column 2

  • Field name - country

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all[1].split("=")[1]
Image Removed
Image Added

Column 3

  • Field name - city

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all[3].split("=")[1]
Image Removed
Image Added

Column 4

  • Field name - surname

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all[4].split("=")[1]
Image Removed
Image Added

Column 5

  • Field name - name

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
all[5].split("=")[1]
Image Removed
Image Added

Column 6

  • Field name - gender

  • Type - String

  • Expression - This is the expression that defines the new column:

Code Block
def g = all[2].split("=")[1]
if (g.equals("male"))
	return "him"
else 
  	return "her"
Image Removed
Image Added

Click Apply to save the configuration.

SlackSink

SlackSin

Image Removed
Image Added

The Slack Sink unit sends a Slack message to a configurable channel. 

Drag a SlackSink unit to the canvas and link the output port out of the Map unit to the input port of this unit.

Image Removed
Image Added

Then, open the configuration options of the unit by double-clicking it and enter the following information:

  • Name - Add a name to the unit (in this example, we called it sendToSlack).

  • Message - This is the message that will be sent to the specified Slack channel. In this example, we will include the following message, referencing the fields defined in the previous unit:
    Dear Devos
    %%﹛name﹜%%﹛surname﹜from %%﹛city﹜in %%﹛country﹜has been added to your system!
    Please join me to welcome %%﹛gender﹜

  • Slack WebHook - Enter the Slack Webhook that references the required Slack workspace and channel where the messages will be sent. Learn more about Slack Webhooks and how to use them here.

Image Removed
Image Added

Click Apply to save the configuration.

Result

Once you have defined the whole Flow, click the Start button to activate it and click the first unit (Tick). If everything is correctly configured, the Slack channel specified in the SlackSink unit through the corresponding Webhook will receive a notification indicating the user info for generated in the my.app table indicated.

This is how the Slack message looks like:

...

📁 Import this Flow

View file
nameSimple ETL.json

Download this Flow in JSON format and import it to your domain clicking using the Import option at the top of the Flow canvas option:

...

Note

Add the required data to make this Flow work

As explained in the Simple ETL configuration section, you must include the my.app table where you want to inject the data and then read it in the DevoSink and DevoSource units to make this flow work. Also, you must indicate the required Slack Webhook in the SlackSink unit.