Simple ETL

Tick

Open

This unit will send an event each time we click it, which will call an external web service added in the next unit.

Drag a Tick unit to the canvas and open its configuration options by double-clicking it.

In the General tab, simply add a name to the unit (in this example, we called it callService).

We don't need to add anything in the Fields tab since we only need to call a request to the external web service added in the following unit upon demand, that is to say, when clicking this unit.

Click Apply to save the configuration.

HttpCall

This unit will call a request to the RandomUser API each time we click the Tick unit added. A list of random users will be added to the response column that will be added to the events.

Drag a HttpCall unit to the canvas and link the output port of the Tick unit to the input port of this unit.

Then, open the configuration options of the unit by double-clicking it and enter the following values in the General tab:

• Name - getUser

• Host - http://randomuser.me

• Path - /api

Click Apply to save the configuration.

JsonParser

This unit will parse the response string generated by the RandomUser API and transform it into a JSON object.

Drag a JsonParser unit to the canvas and link the output port out of the HttpCall unit to the input port of this unit.

Then, open the configuration options of the unit by double-clicking it and enter the following values:

• Name - jsonParser

• In field - Choose the response field generated by the HttpCall unit.

• Out field - Type the name of the JSON field which will be created. In this example, we will call it data.

Click Apply to save the configuration.

Map

This unit will extract the required information from the received JSON objects.

Drag a Map unit to the canvas and link the output port out of the JsonParser unit to the input port of this unit.

Then, open the configuration options of the unit by double-clicking it. In the General tab, simply add a name to the unit (in this example, we called it getFields).

Now go to the Fields to add tab. Click the + icon  in the fields section to add 8 new columns to the events sent:

Column 1

This column extracts all the info from the JSON object. Fro there, we will start extracting the required parts.

• Field name - all

• Type - Map

• Expression - This is the expression that defines the new column:

Column 2

This column extracts the name from the JSON object.

• Field name - name

• Type - String

• Expression - This is the expression that defines the new column:

Column 3

This column extracts the surname from the JSON object.

• Field name - surname

• Type - String

• Expression - This is the expression that defines the new column:

Column 4

This column extracts the gender from the JSON object.

• Field name - gender

• Type - String

• Expression - This is the expression that defines the new column:

Column 5

This column extracts the email from the JSON object.

• Field name - email

• Type - String

• Expression - This is the expression that defines the new column:

Column 6

This column extracts the location from the JSON object.

• Field name - location

• Type - java.lang.Object

• Expression - This is the expression that defines the new column:

Column 7

This column extracts the city from the JSON object.

• Field name - city

• Type - String

• Expression - This is the expression that defines the new column:

Column 8

This column extracts the country from the JSON object.

• Field name - country

• Type - String

• Expression - This is the expression that defines the new column:

Click Apply to save the configuration.

Map

This unit will delete the fields that we don't need from the received events.

Drag another Map unit to the canvas and link the output port out of the previously added Map unit to the input port of this unit.

Then, open the configuration options of the unit by double-clicking it. In the General tab, simply add a name to the unit (in this example, we called it removeGenericFields).

Now go to the Fields to remove tab. Click the Fields field to expand a dropdown menu that contains all the columns in your events. In this case, we will select these ones in order to remove them: response, data, all and location.

Click Apply to save the configuration.

DevoSink

This unit will inject the received data to a Devo my.app table.

Drag a DevoSink unit to the canvas and link the output port out of the second Map unit to the input port of this unit.

Then, open the configuration options of the unit by double-clicking. Enter a Name for the unit (in this example, we called it sendToDevo) and the Table name where you want to send the data. 

Click Apply to save the configuration.

DevoSource

This unit will read the data from the my.app table where we injected the events.

Drag a DevoSource unit to the canvas and open its configuration options by double-clicking it.

Enter a Name for the unit (in this example, readFromDevo)and check off for the Check DB configuration field. In the Query field, enter the query from which you want to get data. In this example, we will use the following:

Also, you can optionally choose a Start time for the query.

Click Apply to save the configuration.

Map

We will use this unit to get the information we want to send through Slack and format it as required.

Drag a Map unit to the canvas and link the output port data of the DevoSource unit to the input port of this unit.

Then, open the configuration options of the unit by double-clicking it. In the General tab, simply add a name to the unit (in this example, we called it messageToEvent).

Now go to the Fields to add tab. Click the + icon in the fields section to add 6 new columns to the events sent, including the required format we will use to send the messages through Slack:

Column 1

• Field name - all

• Type - java.lang.String[]

• Expression - This is the expression that defines the new column:

Column 2

• Field name - country

• Type - String

• Expression - This is the expression that defines the new column:

Column 3

• Field name - city

• Type - String

• Expression - This is the expression that defines the new column:

Column 4

• Field name - surname

• Type - String

• Expression - This is the expression that defines the new column:

Column 5

• Field name - name

• Type - String

• Expression - This is the expression that defines the new column:

Column 6

• Field name - gender

• Type - String

• Expression - This is the expression that defines the new column:

Click Apply to save the configuration.

SlackSin

The Slack Sink unit sends a Slack message to a configurable channel. 

Drag a SlackSink unit to the canvas and link the output port out of the Map unit to the input port of this unit.

Then, open the configuration options of the unit by double-clicking it and enter the following information:

• Name - Add a name to the unit (in this example, we called it sendToSlack).

• Message - This is the message that will be sent to the specified Slack channel. In this example, we will include the following message, referencing the fields defined in the previous unit:
  Dear Devos
%%﹛name﹜%%﹛surname﹜from %%﹛city﹜in %%﹛country﹜has been added to your system!
Please join me to welcome %%﹛gender﹜

• Slack WebHook - Enter the Slack Webhook that references the required Slack workspace and channel where the messages will be sent. Learn more about Slack Webhooks and how to use them here.

Click Apply to save the configuration.