Forward response to HDFS

[ 1 Overview ] [ 2 Parameters ] [ 3 Query using QueryID and forwarding to HDFS ] [ 4 Query using LINQ and forwarding to HDFS ]

Overview

Validate destination first

Before sending data to HDFS, contact Devo customer support to validate the destination.

Using the destination object in a query request, you can have query responses forwarded to your organization's data storage services, including Hadoop Distributed File System (HDFS)-type systems.

To send your response to HDFS, include the destination object in your request and set type to hdfs like this:

"destination": {
 "type":"hdfs"
}

Parameters

With the destination.type set to hdfs, you need to specify some additional parameters specific to HDFS to identify where and how to save the data contained in the response:

Parameter	Description

Parameter	Description
`fs.defaultFS`	Specifies where the HDFS server is listening (for example, `hdfs://172.16.1.17:8020`).
`full path`	Specifies the path where the data will be written (for example, `/tmp/devo/test_05`).
`file`	Specifies the file to which the data will be saved. Data files have the following format: `{file_name}_{node_name}.data`
`dfs.replication`	Defines the default block replication factor. The actual number of replications can be specified when the file is created.

Query using QueryID and forwarding to HDFS

{
    "queryId": "Query_Id_from_web_app",
    "from": 1481471880,
    "to": 1481558280,
    "destination": {
        "type":"hdfs",
        "params": {
            "fs.defaultFS":"hdfs://server:port",
            "dfs.replication":"1",
            "file":"/tmp/devo/test_05"
            ...........
        }
    }
    "mode": {
       "type":"json"
    }   
}

Query using LINQ and forwarding to HDFS

{
    "query": "LinQ_Query",
    "from": 1481471880,
    "to": 1481558280,
    "destination": {
        "type":"hdfs",
        "params": {
            "fs.defaultFS":"hdfs://server:port",
            "dfs.replication":"1",
            "file":"/tmp/devo/test_05"
            ...........
        }
    } 
    "mode": {
       "type":"json"
    }
}

More about HDFS

Learn more about HDFS by reviewing the Hadoop HDFS Architecture Guide.