Page Comparison

These are the steps to follow in order to create a lookup with the Lookups API using a CSV file located in an S3 bucket:

...

Rw step

Locate your Lookup Manager AWS user. For the US, use the following:

arn:aws:iam::175688291360:user/devo-lookups-prod-us

Rw step

Make sure you have access to an S3 bucket in the same AWS region as the Lookup Manager and write down its name. For example lookups-bucket

Rw step

Apply a policy to the bucket which grants the s3:GetObject, the s3:PutObject, and the s3:DeleteObject permissions to the Lookup Manager AWS user in your bucket so that it has permission to download and delete objects from it. See some examples of these policies on the AWS docs.

Rw step

Upload the required CSV file to the bucket, for example, lookup-data.csv. You can upload it to any location you want in the bucket.

Rw step

Get the CSV file S3 URI. You can access the bucket on the AWS console web page using the button Copy S3 URI. This way, you will get a URI that looks like this: s3://lookups-bucket/lookup-data.csv.

...

Rw step

Get the object key from that URI. The structure of the S3 URI is the following: s3://<bucket-name>/<object-key>. In this case, our key would be lookup-data.csv.

Rw step

...

Table of Contents

minLevel	1
maxLevel	2
outline	false
type	flat
separator	brackets
printable	false

Prerequisites

An existing S3 bucket in your organization's AWS Account
An IAM User in your organization's AWS Account with access to your existing S3 bucket.

Procedure

These are the steps to follow in order to create a lookup with the Lookups API using a CSV file located in an S3 bucket.

Rw ui steps macro

Rw step

Upload a CSV file to the bucket. In this example, ip-dst_misp_example.csv. You can upload it to any location you want in the bucket. In the following example, we’ve uploaded it under the root folder of the bucket, in a folder called lookups_csv.

Image Added

Rw step

Navigate to the Objects tab inside your bucket and locate your CSV. Click it to display its properties.

Rw step

Get the Key and the AWS Region of the CSV file by clicking on the button in the Key section.

Image Added

Rw step

Get or create an Access Key for your IAM User and obtain its Secret Key so the Lookups Manager can access your AWS Account.

Rw step

Create a JSON payload to be added to your lookup creation request to the API. It must include a Source object which does not include a query attribute, but includes a fileProvider one with the name of the bucket, the access key and the secret key of the AWS IAM User, the AWS region where the bucket has been created and the key of the file. For example:

{ "id":

Code Block

{
  "visibility": "creator-only",
  "recipe": {
    "recipeType": "once",
    "source": {
      "columns": [
        {
          "name": "ipAddr",
          "from": 0,
          "type": "IP4"
        },
        {
    "creator": "lookups_domain",     "name": "LocationsLookupfirstSeen",
   },   "visibility": "creator-only",   "recipefrom": {1,
    "recipeType": "once",     "sourcetype": "STRING"
{       "columns": [ },
        {
          "name": "IDlastSeen",
          "from": 02,
          "type": "INT4STRING"
        },
        {
          "name": "Locationcomment",
          "from": 3,
          "type": 1,
    "STRING"
        }
      ],
      "skipPreface": null,
      "typehasHeader": "STRING" true,
      "skipEmptyLines": false,
  }    "fileProvider": {
 ],       "skipPrefacebucketName": null,"devo-lookups-client1-stage",
        "hasHeaderkeyName": false,
      "skipEmptyLines": false,
"lookups_csv/ip-dst_misp_example.csv",
        "fileProvideraccessKey": {"abcdefghijkl",
        "bucketNamesecretKey": "lookups-bucketTg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ",
        "keyNameregion": "lookup-data.csveu-west-1",
        "transferOwnership": true
      }
    },
    "lookupType": {
      "type": "normal"
    },
    "append": false,
    "key": {
      "type": "column",
      "column": "IDipAddr"
    },
    "columnFilter": [": [
      "ipAddr",
      "firstSeen",
      "IDlastSeen",
      "Locationcomment"
    ],
    "contribution": {
      "type": "add"
    },
    "requiresDate": false
  }
}

Rw step

Create a HTTP POST or PUT request with the created payload:

Code Block

curl --location --request POST 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/LocationsLookupIPsLookup/deploy-config' \
--header 'Authorization: Bearer <your-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "id": {
    "creator": "lookups_domain",
    "name": "LocationsLookup"
  },/json' \
--data-raw '{
  "visibility": "creator-only",
  "recipe": {
    "recipeType": "once",
    "source": {
      "columns": [
        {
          "name": "IDipAddr",
          "from": 0,
          "type": "INT4IP4"
        },
        {
          "name": "Location",
          "from": 1,
          "type": "STRING"firstSeen",
        }  "from": 1,
   ],       "skipPrefacetype": null,
      "hasHeaderSTRING":
false,       "skipEmptyLines": false},
       "fileProvider": {
          "bucketNamename": "lookups-bucketlastSeen",
          "keyNamefrom": "lookup-data.csv", 2,
          "transferOwnershiptype": "STRING"
true        },
     },   {
  "lookupType": {       "typename": "normalcomment",
     },     "appendfrom": false3,
          "keytype": {
 "STRING"
        }
     "type": "column" ],
      "columnskipPreface": "ID"null,
    },
    "columnFilterhasHeader": [true,
      "IDskipEmptyLines": false,
      "LocationfileProvider": {
   ],     "contributionbucketName": {"devo-lookups-client1-stage",
        "typekeyName": "add"
lookups_csv/ip-dst_misp_example.csv",
   },     "requiresDateaccessKey": false"abcdefghijkl",
  } }

You should get a successful response like the following:

Code Block

{     "typesecretKey": "LookupCreationResponseTg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ",
        "cidregion": "ba9ac1c28205eu-west-1",
        "codetransferOwnership": 201,false
    "context": null, }
   "id": "46a68695-543d-11ed-b24b-b102fd5ab44d" },
    "msglookupType": "Lookup{
sent to creation",    "type": "lookupDeployConfignormal":
{    },
    "idappend": {false,
    "key": {
      "creatortype": "lookups_domain",
     column",
      "namecolumn": "LocationsLookupipAddr"
    },
   }, "columnFilter": [
      "visibility": "creator-only",
 ipAddr",
      "recipefirstSeen":,
{      "lastSeen",
      "recipeTypecomment":
"once",        ],
    "sourcecontribution": {
      "type": "add"
    },
    "columnsrequiresDate": [false
  }
}'

You should get a response similar to this:

Code Block

{
  "type": "LookupCreationResponse",
  "cid": "65574b8c3463",
  "code": 201,
  "context": null,
{    "id": "baa1b1ef-6430-11ed-9fc0-efb84bae0957",
  "msg": "Lookup sent to creation. You can check the lookup status using the provided     "name": "ID",
         id: /lookup/{domain}/{name}/job/{id}",
  "lookupDeployConfig": {
    "id": {
        "fromcreator": 0"lookups_domain",
      "name": "IPsLookup"
          },
     "typevisibility": "INT4creator-only",
                "recipe": {
   },   "recipeType": "once",
      "source": {
        {"columns": [
          {
            "name": "LocationipAddr",
           
            "from": 10,
            "type": "IP4"
          "type": "STRING"},
              {
      }      "name": "firstSeen",
         ],   "from": 1,
            "skipPrefacetype": null,"STRING"
          },
     "hasHeader": false,    {
            "skipEmptyLinesname": false"lastSeen",
            "from": 2,
     "fileProvider": {      "type": "STRING"
          },
  "bucketName": "lookups-bucket",       {
             "keyNamename": "lookup-data.csvcomment",
            "from": 3,
            "transferOwnershiptype": true"STRING"
          }
        }],
        "skipPreface": null,
        "queryhasHeader": null
true,
        "skipEmptyLines": false,
 },       "fileProvider": {
    "lookupType": {     "bucketName": "devo-lookups-client1-stage",
          "typekeyName": "normallookups_csv/ip-dst_misp_example.csv",
   
            "instantPolicyaccessKey": null"abcdefghijkl",
          "secretKey": "Tg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ",
    "instant": null,     "region": "eu-west-1",
          "columnNametransferOwnership": null
   false
        },

           "appendquery": false,null
      },
      "keylookupType": {
        "type": "normal",
        "columnsinstantPolicy": []null,
        "instant": null,
        "columncolumnName": "ID",null
      },
         "typeappend": false,
"column"      "key": {
     },   "columns": [],
        "columnFiltercolumn": [
      "ipAddr",
        "type": "IDcolumn",
      },
      "columnFilter": [
 "Location"       "ipAddr",
     ],   "firstSeen",
         "contributionlastSeen":,
{        "comment"
        "type": "add"],
      "contribution": {
        "nametype": null
"add",
        "name": null
 },      },
      "secondaryIndexes": null,
     
      "refreshMillis": null,
    
       "startMillis": null,
            "requiresDate": false
   
    },
   
    "notifyStatus": null
    }
}

Rw step

You can check for the status state of the lookup creation/update querying the /<domain>/<lookup>/job/<id> You can start using the lookup once and if you see the message Lookup ready to be executedto be executed, you can start using the lookup.

Code Block

curl --location --request GET 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/LocationsLookupIPsLookup/job/46a68695baa1b1ef-543d6430-11ed-b24b9fc0-b102fd5ab44defb84bae0957' \
--header 'Authorization: Bearer <your-token>'
{
    "type": "LookupJobListResponseLookupJobStagesListResponse",
    "cid": "9aaa3cd93327b5c3f2981537",
    "code": 200,
    "context": null,
    "id": "ee2bd9f647c2b160-543e6431-11ed-b24b9fc0-9bb4e40c2d97cd58ebae0bc3",
    "msg": "Lookup jobs",
    "jobs": [
        {
            "eventdate": "2022-1011-24T1214T15:3126:3649.93414",
            "domain": "lookups_domaindomain",
            "lookup": "IPsLookup",
            "lookupmsg": "LocationsLookupLookup successfully created",
            "msgcode": "Lookup successfully createdcreate.ok"
        },
        {
            "eventdate": "2022-1011-24T1214T15:3227:0711.294767",
            "domain": "lookups_domainsdomain",
            "lookup": "LocationsLookupIPsLookup",
            "msg": "Lookup ready to be executed"
            "code": "deploy.ok"
        }
    ],
    "nextPageToken": 16666146966081668439608968
}

Versions Compared

Old Version 6

New Version Current

Key

Prerequisites

Procedure