Table of Contents | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Prerequisites
An existing S3 bucket in your organization's AWS Account
An IAM User in your organization's AWS Account with access to your existing S3 bucket.
Procedure
These are the steps to follow in order to create a lookup with the Lookups API using a CSV file located in an S3 bucket:.
Rw ui steps macro | |
---|---|
Locate your Lookup Manager AWS user. Use the required one according to your environment: | |
Devo environment | Lookup Manager AWS user |
arn:aws:iam::175688291360:user/devo-lookups-prod-us | |
arn:aws:iam::175688291360:user/devo-lookups-prod-eu | |
arn:aws:iam::175688291360:user/devo-lookups-pro-ca | |
arn:aws:iam::175688291360:user/devo-lookups-prod-apac |
Rw step |
---|
Make sure you have access to a S3 bucket in the same AWS region as the Lookup Manager and write down its name. For this example, we are going to use devo-lookups-client1-stage
.
Rw step |
---|
Access the Permissions page in the bucket.
Rw step |
---|
Click the Edit button in the Block public access section.
Rw step |
---|
Uncheck all the options in the Block public access section and click the Save changes button.
Rw step |
---|
Click the Edit button in the Bucket policy section to add a policy that allows the user to perform the actions s3:GetObject
(download your CSVs from the bucket), and s3:DeleteObject
(delete the CSVs in case the transferOwnership
argument is set to true
in the request). Then, click Save changes.
In the following image, we are granting permissions to the bucket specified in the Statement.Resource
parameter to the user specified in the Statement.Principal.AWS
parameter.
Rw step |
---|
Then, upload a CSV file to the bucket (in this example, ip-dst_misp_example.csv
). You can upload it to wherever Upload a CSV file to the bucket. In this example, ip-dst_misp_example.csv
. You can upload it to any location you want in the bucket. In the following example, it has been uploaded to a location we’ve uploaded it under the root folder of the bucket, in a folder called lookups_csv
.
Rw step |
---|
Navigate to the Objects tab inside your bucket and locate your CSV. Click it to display its properties.
Rw step |
---|
Get the keyGet the Key
and the AWS Region
of the CSV file by clicking on the button in the Key section.
Rw step |
---|
Get or create an Access Key
for your IAM User and obtain its Secret Key
so the Lookups Manager can access your AWS Account.
Rw step |
---|
Create a JSON payload to be added to your lookup creation request to the API request. It must include a Source
object without the query
attribute, but including the fileProvider
parameter with the name of the bucket which does not include a query
attribute, but includes a fileProvider
one with the name of the bucket, the access key and the secret key of the AWS IAM User, the AWS region where the bucket has been created and the key of the file. For example:
example:
Code Block |
---|
{ "visibility": "creator-only", "recipe": { "recipeType": "once", "source": { "columns": [ { "name": "ipAddr", "from": 0, "type": "IP4" }, { "name": "firstSeen", "from": 1, "type": "STRING" }, { "name": "lastSeen", "from": 2, "type": "STRING" }, { "name": "comment", "from": 3, "type": "STRING" } ], "skipPreface": null, null, "hasHeader": true, "skipEmptyLines": false, "fileProvider": { "hasHeaderbucketName": true,"devo-lookups-client1-stage", "skipEmptyLineskeyName": false,"lookups_csv/ip-dst_misp_example.csv", "fileProvideraccessKey": {"abcdefghijkl", "bucketNamesecretKey": "devo-lookups-client1-stageTg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ", "keyNameregion": "lookups_csv/ip-dst_misp_example.csveu-west-1", "transferOwnership": true } }, "lookupType": { "type": "normal" }, "append": false, "key": { "type": "column", "column": "ipAddr" }, "columnFilter": [ "ipAddr", "firstSeen", "lastSeen", "comment" ], "contribution": { "type": "add" }, "requiresDate": false } } |
Rw step |
---|
Create a HTTP POST or PUT request with the created payload:
Code Block |
---|
curl --location --request POST 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/IPsLookup/deploy-config' \ --header 'Authorization: Bearer <your-token>' \ --header 'Content-Type: application/json' \ --data-raw '{ "id": { "creator": "lookups_domain", "name": "IPsLookup" },: application/json' \ --data-raw '{ "visibility": "creator-only", "recipe": { "recipeType": "once", "source": { "columns": [ { "name": "ipAddr", "from": 0, "type": "IP4" }, { "name": "firstSeen", "from": 1, "type": "STRING" }, { "name": "lastSeen", "from": 2, "type": "STRING" }, { "name": "comment", "from": 3, "type": "STRING" } ], "skipPreface": null, "hasHeader": true,": true, "skipEmptyLines": false, "fileProvider": { "bucketName": "devo-lookups-client1-stage", "skipEmptyLineskeyName": false"lookups_csv/ip-dst_misp_example.csv", "fileProvideraccessKey": {"abcdefghijkl", "bucketNamesecretKey": "devo-lookups-client1-stageTg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ", "keyNameregion": "lookups_csv/ip-dst_misp_example.csveu-west-1", "transferOwnership": false } }, "lookupType": { "type": "normal" }, "append": false, "key": { "type": "column", "column": "ipAddr" }, "columnFilter": [ "ipAddr", "firstSeen", "lastSeen", "comment" ], "contribution": { "type": "add" }, "requiresDate": false } }' |
You should get a successful response of the likes ofresponse similar to this:
Code Block |
---|
{ "type": "LookupCreationResponse", "cid": "65574b8c3463", "code": 201, "context": null, "id": "baa1b1ef-6430-11ed-9fc0-efb84bae0957", "msg": "Lookup sent to creation. You can check the lookup status using the provided id: /lookup/{domain}/{name}/job/{id}", "lookupDeployConfig": { "id": { "creator": "lookups_domain", "name": "IPsLookup" }, "visibility": "creator-only", "recipe": { "recipeType": "once", "source": { "columns": [ { "name": "ipAddr", "from": 0, "type": "IP4" }, { "name": "firstSeen", "from": 1, "type": "STRING" }, { "name": "lastSeen", "from": 2, "type": "STRING" }, { { "name": "comment", "from": 3, "nametype": "commentSTRING", } "from": 3, ], "typeskipPreface": "STRING"null, "hasHeader": true, } "skipEmptyLines": ]false, "skipPrefacefileProvider": { null, "bucketName": "hasHeader": true,"devo-lookups-client1-stage", "skipEmptyLineskeyName": false"lookups_csv/ip-dst_misp_example.csv", "fileProvideraccessKey": {"abcdefghijkl", "bucketNamesecretKey": "devo-lookups-client1-stageTg4T0aGKvd/aaaaaaaWWQv3Vs0kS15tpn3kbd0V2UZ", "keyNameregion": "lookups_csv/ip-dst_misp_example.csveu-west-1", "transferOwnership": false }, "query": null }, "lookupType": { "type": "normal", "instantPolicy": null, "instant": null, "columnName": null }, "append": false, "key": { "columns": [], "column": "ipAddr", "type": "column" }, "columnFilter": [ "ipAddr", "firstSeen", "lastSeen", "comment" ], "contribution": { "type": "add", "name": null }, "secondaryIndexes": null, "refreshMillis": null, "startMillis": null, "requiresDate": false }, "notifyStatus": null } } |
Rw step |
---|
You can check for the status state of the creation/update querying the /<domain>/<lookup>/job/<id>
. If and if you see the message Lookup ready to be executed
, you can start using the lookup.
Code Block |
---|
curl --location --request GET 'https://<devo-apis-host>/lookup-api/lookup/lookups_domain/IPsLookup/job/baa1b1ef-6430-11ed-9fc0-efb84bae0957' \ --header 'Authorization: Bearer <your-token>' { "type": "LookupJobListResponseLookupJobStagesListResponse", "cid": "b5c3f2981537", "code": 200, "context": null, "id": "47c2b160-6431-11ed-9fc0-cd58ebae0bc3", "msg": "Lookup jobs", "jobs": [ { "eventdate": "2022-11-14T15:26:49.14", "domain": "lookups_domain", "lookup": "IPsLookup", "msg": "Lookup successfully created" "code": "create.ok" }, { "eventdate": "2022-11-14T15:27:11.767", "domain": "lookups_domain", "lookup": "IPsLookup", "msg": "Lookup ready to be executed" "code": "deploy.ok" } ], "nextPageToken": 1668439608968 } |