Forward response to HDFS
- Juan Tomás Alonso Nieto (Deactivated)
- Borja Moro Moreno (Unlicensed)
Overview
Validate destination first
Before sending data to HDFS, contact Devo customer support to validate the destination.
Using the destination object in a query request, you can have query responses forwarded to your organization's data storage services, including Hadoop Distributed File System (HDFS)-type systems.
To send your response to HDFS, include the destination object in your request and set type to hdfs like this:
"destination": {
"type":"hdfs"
}Parameters
With the destination.type set to hdfs, you need to specify some additional parameters specific to HDFS to identify where and how to save the data contained in the response:
Parameter | Description |
|---|---|
| Specifies where the HDFS server is listening (for example, |
| Specifies the path where the data will be written (for example, |
| Specifies the file to which the data will be saved. Data files have the following format: |
| Defines the default block replication factor. The actual number of replications can be specified when the file is created. |
Query using QueryID and forwarding to HDFS
{
"queryId": "Query_Id_from_web_app",
"from": 1481471880,
"to": 1481558280,
"destination": {
"type":"hdfs",
"params": {
"fs.defaultFS":"hdfs://server:port",
"dfs.replication":"1",
"file":"/tmp/devo/test_05"
...........
}
}
"mode": {
"type":"json"
}
}Query using LINQ and forwarding to HDFS
{
"query": "LinQ_Query",
"from": 1481471880,
"to": 1481558280,
"destination": {
"type":"hdfs",
"params": {
"fs.defaultFS":"hdfs://server:port",
"dfs.replication":"1",
"file":"/tmp/devo/test_05"
...........
}
}
"mode": {
"type":"json"
}
}More about HDFS
Learn more about HDFS by reviewing the Hadoop HDFS Architecture Guide.