Page Comparison

These are the steps to follow in order to create a lookup with the Lookups API using a CSV file located in an S3 bucket:

...

Rw step

Locate your Lookup Manager AWS user. Use the required one according to your environment:

...

Devo environment

...

Lookup Manager AWS user

...

us.devo.com

...

arn:aws:iam::175688291360:user/devo-lookups-prod-us

...

eu.devo.com

...

arn:aws:iam::175688291360:user/devo-lookups-prod-eu

...

ca.devo.com

...

arn:aws:iam::175688291360:user/devo-lookups-pro-ca

...

apac.devo.com

...

arn:aws:iam::175688291360:user/devo-lookups-prod-apac

Rw step

Make sure you have access to a S3 bucket in the same AWS region as the Lookup Manager and write down its name. For this example, we are going to use devo-lookups-client1-stage.

Rw step

Access the Permissions page in the bucket.

Rw step

Click the Edit button in the Block public access section.

...

Rw step

Uncheck all the options in the Block public access section and click the Save changes button.

...

Rw step

Click the Edit button in the Bucket policy section to add a policy that allows the user to perform the actions s3:GetObject (download your CSVs from the bucket), and s3:DeleteObject (delete the CSVs in case the transferOwnership argument is set to true in the request). Then, click Save changes.

In the following image, we are granting permissions to the bucket specified in the Statement.Resource parameter to the user specified in the Statement.Principal.AWS parameter.

...

Rw step

...

.

Prerequisites

An existing S3 bucket in your organization's AWS Account
An IAM User in your organization's AWS Account with access to your existing S3 bucket.

Rw ui steps macro

Rw step

Upload a CSV file to the bucket. In this example, ip-dst_misp_example.csv). You can upload it to wherever any location you want in the bucket. In the following example, it has been uploaded to a location we’ve uploaded it under the root folder of the bucket, in a folder called lookups_csv.

Image Removed

Image Added

Rw step

Navigate to the Objects tab inside your bucket and locate your CSV. Click it to display its properties.

Rw step

Get the Key and the key AWS Region of the CSV file by clicking the button in the Key section.

Image Removed

clicking on the button in the Key section.

Image Added

Rw step

Get or create an Access Key for your IAM User and obtain its Secret Key so the Lookups Manager can access your AWS Account.

Rw step

Create a JSON payload to be added to your lookup creation request to the API request. It must include a Source object without the which does not include a query attribute, but including the includes a fileProvider parameter one with the name of the bucket the bucket, the access key and the secret key of the AWS IAM User, the AWS region where the bucket has been created and the key of the file. For example:

Code Block

{
  "id": {
    "creator": "lookups_domain",
    "name": "IPsLookup"
  },
  "visibility": "creator-only",
  "recipe": {
    "recipeType": "once",
    "source": {
      "columns": [
        {
          "name": "ipAddr",
          "from": 0,
          "type": "IP4"
        },
        {
          "name": "firstSeen",
          "from": 1,
          "type": "STRING"
        },
        {
          "name": "lastSeen",
          "from": 2,
          "type": "STRING"
        },
        {
          "name": "comment",
          "from": 3,
          "type": "STRING"
        }
      ],
      "skipPreface": null,
      "hasHeader": true,
      "skipEmptyLines": false,
      "fileProvider": {
        "bucketName": "devo-lookups-client1-stage",
        "keyName": "lookups_csv/ip-dst_misp_example.csv.csv",
        "accessKey": "abcdefghijkl",
        "secretKey": "Tg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ",
        "region": "eu-west-1",
        "transferOwnership": true
      }
    },
    "lookupType": {
      "type": "normal"
    },
    "append": false,
    "key": {
      "type": "column",
      "column": "ipAddr"
    },
    "columnFilter": [
      "ipAddr",
      "firstSeen",
      "lastSeen",
      "comment"
    ],
    "contribution": {
      "type": "add"
    },
    "requiresDate": false
  }
}

Rw step

Create a HTTP POST or PUT request with the created payload:

Code Block

curl --location --request POST 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/IPsLookup/deploy-config' \
--header 'Authorization: Bearer <your-token>' \
--header 'Content-Type: application/json' \
--data-raw '{-raw '{
  "id": {
    "creator": "lookups_domain",
    "name": "IPsLookup"
  },
  "visibility": "creator-only",
  "idrecipe": {
    "creatorrecipeType": "lookups_domainonce",
    "namesource": {
      "IPsLookup"columns": [
        {
       },   "visibilityname": "creator-only",ipAddr",
          "recipefrom": {0,
    "recipeType": "once",     "sourcetype": {"IP4"
       "columns": [ },
        {
          "name": "ipAddrfirstSeen",
          "from": 01,
          "type": "IP4STRING"
        },
        {
          "name": "firstSeenlastSeen",
          "from": 12,
          "type": "STRING"
        },
        {
          "name": "lastSeencomment",
          "from": 23,
          "type": "STRING"
        },
       ],
{           "nameskipPreface": "comment",
   null,
      "fromhasHeader": 3true,
   
      "typeskipEmptyLines": "STRING"false,
        }"fileProvider": {
      ],       "skipPrefacebucketName": null,"devo-lookups-client1-stage",
        "hasHeaderkeyName": true, "lookups_csv/ip-dst_misp_example.csv",
        "skipEmptyLinesaccessKey": false,
      "fileProviderabcdefghijkl": {,
        "bucketNamesecretKey": "devo-lookups-client1-stageTg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ",
        "keyNameregion": "lookups_csv/ip-dst_misp_example.csveu-west-1",
        "transferOwnership": false
      }
    },
    "lookupType": {
      "type": "normal"
    },
    "append": false,
    "key": {
      "type": "column",
      "column": "ipAddr"
    },
    "columnFilter": [
      "ipAddr",
      "firstSeen",
      "lastSeen",
      "comment"
    ],
    "contribution": {
      "type": "add"
    },
    "requiresDate": false
  }
}'

rw-step

You should get a successful response of the likes ofresponse similar to this:

Code Block

{
  "type": "LookupCreationResponse",
  "cid": "65574b8c3463",
  "code": 201,
  "context": null,
  "id": "baa1b1ef-6430-11ed-9fc0-efb84bae0957",
  "msg": "Lookup sent to creation. You can check the lookup status using the provided id: /lookup/{domain}/{name}/job/{id}",
  "lookupDeployConfig": {
    "id": {
      "creator": "lookups_domain",
      "name": "IPsLookup"
    },
    "visibility": "creator-only",
    "recipe": {
      "recipeType": "once",
      "source": {
        "columns": [
          {
            "name": "ipAddr",
            "from": 0,
            "type": "IP4"
          },
          {
            "name": "firstSeen",
            "from": 1,
            "type": "STRING"
          },
          {
            "name": "lastSeen",
            "from": 2,
            "type": "STRING"
          },
          {
            "name": "comment",
            "from": 3,
            "type": "STRING"
          }
        ],
        "skipPreface": null,
        "hasHeader": true,
        "skipEmptyLines": false,
        "fileProvider": {
        "hasHeader": true,    "bucketName": "devo-lookups-client1-stage",
          "skipEmptyLineskeyName": false,"lookups_csv/ip-dst_misp_example.csv",
          "fileProvideraccessKey": {"abcdefghijkl",
          "bucketNamesecretKey": "devo-lookups-client1-stageTg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ",
          "keyNameregion": "lookups_csv/ip-dst_misp_example.csveu-west-1",
          "transferOwnership": false
        },
        "query": null
      },
      "lookupType": {
        "type": "normal",
        "instantPolicy": null,
        "instant": null,
        "columnName": null
      },
      "append": false,
      "key": {
        "columns": [],
        "column": "ipAddr",
        "type": "column"
      },
      "columnFilter": [
        "ipAddr",
        "firstSeen",
        "lastSeen",
        "comment"
      ],
      "contribution": {
        "type": "add",
        "name": null
      },
      "secondaryIndexes": null,
      "refreshMillis": null,
      "startMillis": null,
      "requiresDate": false
    },
    "notifyStatus": null
  }
}

Rw step

You can check for the status state of the creation/update querying the /<domain>/<lookup>/job/<id> . If and if you see the message Lookup ready to be executed, you can start using the lookup.

Code Block

curl --location --request GET 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/IPsLookup/job/baa1b1ef-6430-11ed-9fc0-efb84bae0957' \
--header 'Authorization: Bearer <your-token>'
{
    "type": "LookupJobListResponseLookupJobStagesListResponse",
    "cid": "b5c3f2981537",
    "code": 200,
    "context": null,
    "id": "47c2b160-6431-11ed-9fc0-cd58ebae0bc3",
    "msg": "Lookup jobs",
    "jobs": [
        {
            "eventdate": "2022-11-14T15:26:49.14",
            "domain": "lookups_domain",
            "lookup": "IPsLookup",
            "msg": "Lookup successfully created"
            "code": "create.ok"
        },
        {
            "eventdate": "2022-11-14T15:27:11.767",
            "domain": "lookups_domain",
            "lookup": "IPsLookup",
            "msg": "Lookup ready to be executed"
            "code": "deploy.ok"
        }
    ],
    "nextPageToken": 1668439608968
}

Versions Compared

Old Version 9

New Version 10

Key

Prerequisites