Versions Compared

Old Version 8

changes.mady.by.user Juan Tomás Alonso Nieto

Saved on

compared with

New Version 9

changes.mady.by.user Juan Tomás Alonso Nieto

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Rw ui steps macro
Rw step

Locate your Lookup Manager AWS user. Use the required one according to your environment:

Devo environment

Lookup Manager AWS user

us.devo.com

arn:aws:iam::175688291360:user/devo-lookups-prod-us

eu.devo.com

arn:aws:iam::175688291360:user/devo-lookups-prod-eu

ca.devo.com

arn:aws:iam::175688291360:user/devo-lookups-pro-ca

apac.devo.com

arn:aws:iam::175688291360:user/devo-lookups-prod-apac

Rw step

Make sure you have access to a S3 bucket in the same AWS region as the Lookup Manager and write down its name. For this example, we are going to use devo-lookups-client1-stage.

Rw step

Access the Permissions page in the bucket.

Rw step

Click the Edit button in the Block public access section.

Image RemovedImage Added
Rw step

Uncheck all the options in the Block public access section and click the Save changes button.

Image RemovedImage Added
Rw step

Click the Edit button in the Bucket policy section to add a policy that allows the user to perform the actions s3:GetObject (download your CSVs from the bucket), and s3:DeleteObject (delete the CSVs in case the transferOwnership argument is set to true in the request). Then, click Save changes.

In the following image, we are granting permissions to the bucket specified in the Statement.Resource parameter to the user specified in the Statement.Principal.AWS parameter.

Image RemovedImage Added
Rw step

Then, upload a CSV file to the bucket (in this example, ip-dst_misp_example.csv). You can upload it to wherever location you want in the bucket. In the following example, it has been uploaded to a location under the root folder of the bucket called lookups_csv.

Image RemovedImage Added
Rw step

Navigate to the Objects tab inside your bucket and locate your CSV. Click it to display its properties.

Rw step

Get the keyof the CSV file by clicking the button in the Key section.

Image RemovedImage Added
Rw step

Create a JSON payload to be added to your API request. It must include a Source object without the query attribute, but including the fileProvider parameter with the name of the bucket and the key of the file. For example:

Code Block
{
  "id": {
    "creator": "lookups_domain",
    "name": "IPsLookup"
  },
  "visibility": "creator-only",
  "recipe": {
    "recipeType": "once",
    "source": {
      "columns": [
        {
          "name": "ipAddr",
          "from": 0,
          "type": "IP4"
        },
        {
          "name": "firstSeen",
          "from": 1,
          "type": "STRING"
        },
        {
          "name": "lastSeen",
          "from": 2,
          "type": "STRING"
        },
        {
          "name": "comment",
          "from": 3,
          "type": "STRING"
        }
      ],
      "skipPreface": null,
      "hasHeader": true,
      "skipEmptyLines": false,
      "fileProvider": {
        "bucketName": "devo-lookups-client1-stage",
        "keyName": "lookups_csv/ip-dst_misp_example.csv",
        "transferOwnership": true
      }
    },
    "lookupType": {
      "type": "normal"
    },
    "append": false,
    "key": {
      "type": "column",
      "column": "ipAddr"
    },
    "columnFilter": [
      "ipAddr",
      "firstSeen",
      "lastSeen",
      "comment"
    ],
    "contribution": {
      "type": "add"
    },
    "requiresDate": false
  }
}
Rw step

Create a HTTP POST or PUT request with the created payload:

Code Block
curl --location --request POST 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/IPsLookup/deploy-config' \
--header 'Authorization: Bearer <your-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "id": {
    "creator": "lookups_domain",
    "name": "IPsLookup"
  },
  "visibility": "creator-only",
  "recipe": {
    "recipeType": "once",
    "source": {
      "columns": [
        {
          "name": "ipAddr",
          "from": 0,
          "type": "IP4"
        },
        {
          "name": "firstSeen",
          "from": 1,
          "type": "STRING"
        },
        {
          "name": "lastSeen",
          "from": 2,
          "type": "STRING"
        },
        {
          "name": "comment",
          "from": 3,
          "type": "STRING"
        }
      ],
      "skipPreface": null,
      "hasHeader": true,
      "skipEmptyLines": false,
      "fileProvider": {
        "bucketName": "devo-lookups-client1-stage",
        "keyName": "lookups_csv/ip-dst_misp_example.csv",
        "transferOwnership": false
      }
    },
    "lookupType": {
      "type": "normal"
    },
    "append": false,
    "key": {
      "type": "column",
      "column": "ipAddr"
    },
    "columnFilter": [
      "ipAddr",
      "firstSeen",
      "lastSeen",
      "comment"
    ],
    "contribution": {
      "type": "add"
    },
    "requiresDate": false
  }
}'
Rw step

You should get a successful response of the likes of:

Code Block
{
  "type": "LookupCreationResponse",
  "cid": "65574b8c3463",
  "code": 201,
  "context": null,
  "id": "baa1b1ef-6430-11ed-9fc0-efb84bae0957",
  "msg": "Lookup sent to creation. You can check the lookup status using the provided id: /lookup/{domain}/{name}/job/{id}",
  "lookupDeployConfig": {
    "id": {
      "creator": "lookups_domain",
      "name": "IPsLookup"
    },
    "visibility": "creator-only",
    "recipe": {
      "recipeType": "once",
      "source": {
        "columns": [
          {
            "name": "ipAddr",
            "from": 0,
            "type": "IP4"
          },
          {
            "name": "firstSeen",
            "from": 1,
            "type": "STRING"
          },
          {
            "name": "lastSeen",
            "from": 2,
            "type": "STRING"
          },
          {
            "name": "comment",
            "from": 3,
            "type": "STRING"
          }
        ],
        "skipPreface": null,
        "hasHeader": true,
        "skipEmptyLines": false,
        "fileProvider": {
          "bucketName": "devo-lookups-client1-stage",
          "keyName": "lookups_csv/ip-dst_misp_example.csv",
          "transferOwnership": false
        },
        "query": null
      },
      "lookupType": {
        "type": "normal",
        "instantPolicy": null,
        "instant": null,
        "columnName": null
      },
      "append": false,
      "key": {
        "columns": [],
        "column": "ipAddr",
        "type": "column"
      },
      "columnFilter": [
        "ipAddr",
        "firstSeen",
        "lastSeen",
        "comment"
      ],
      "contribution": {
        "type": "add",
        "name": null
      },
      "secondaryIndexes": null,
      "refreshMillis": null,
      "startMillis": null,
      "requiresDate": false
    },
    "notifyStatus": null
  }
}
Rw step

You can check the status of the creation/update querying /<domain>/<lookup>/job/<id> . If you see the message Lookup ready to be executed, you can start using the lookup.

Code Block
curl --location --request GET 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/IPsLookup/job/baa1b1ef-6430-11ed-9fc0-efb84bae0957' \
--header 'Authorization: Bearer <your-token>'
{
    "type": "LookupJobListResponse",
    "cid": "b5c3f2981537",
    "code": 200,
    "context": null,
    "id": "47c2b160-6431-11ed-9fc0-cd58ebae0bc3",
    "msg": "Lookup jobs",
    "jobs": [
        {
            "eventdate": "2022-11-14T15:26:49.14",
            "domain": "lookups_domain",
            "lookup": "IPsLookup",
            "msg": "Lookup successfully created"
        },
        {
            "eventdate": "2022-11-14T15:27:11.767",
            "domain": "lookups_domain",
            "lookup": "IPsLookup",
            "msg": "Lookup ready to be executed"
        }
    ],
    "nextPageToken": 1668439608968
}

...