Page Comparison

...

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

1. In the Collector Server GUI, access the domain in which you want this instance to be created

2. Click Add Collector and find the one you wish to add.

3. In the Version field, select the latest value.

4. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).

5. In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.

6. In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

{
  "global_overrides": {
    "debug": false
  },
  "inputs": {
    "snowflake": {
      "id": "<input_id>",
      "enabled": <input_status>,
│   ├── <your_domain>.key  "warehouse": "<warehouse>",
     │ "pager": <pager>,
└── <your_domain>.crt     "credentials": {
  ├── state/     "username": "<username>",
  └── config/     "password": "<password>",
       └── config.yaml 

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Image Removed

Editing the config.yaml file

globals:
  debug: false
  id: "<collector_id>"
  name: "<collector_name>"
  persistence:
    type: filesystem
    config: "account_identifier": "<account_identifier>"
      },
      "services": {
        "access_history": {
           directory_name: state
outputs:
  devo_us_1:"historic_date_utc": "<historic_date_utc>",
          type"request_period_in_seconds": devo_platform<request_period_in_seconds>,
    config:       address"role": "<devo_address><role>"
        port:},
 443       type"login_history": SSL{
          chain"historic_date_utc": "<chain<historic_date_filename>utc>",
         cert: "<cert_filename>"request_period_in_seconds": <request_period_in_seconds>,
      key    "role": "<key_filename><role>"
inputs:   snowflake:     id: "<input_id>"},
        "sessions": {
    enabled: <input_status>     warehouse"historic_date_utc": "<warehouse><historic_date_utc>",
    pager: <pager>     credentials:
   "request_period_in_seconds": <request_period_in_seconds>,
  username: "<username>"       password"role": "<password><role>"
      account_identifier: "<account_identifier>"  },
    services:    "custom_sql": {
 access_history:         historic_date_utc"types": ["<historiccustom_date_utc>"db"],
          "request_period_in_seconds": <request_period_in_seconds>,
        role  "sql_query": "<role><sql_query>"
,
     login_history:     "database": "<database>",
  historic_date_utc: "<historic_date_utc>"       "schema":  request_period_in_seconds: <request_period_in_seconds>"<schema>",
          "role": "<role>"
  ,
   sessions:         historic_date_utc: "<historicoverride_datedevo_utc>"
        request_period_in_seconds: <request_period_in_seconds>
        role: "<role>tag": "<override_tag_value>"
      custom_sql:  }
      types:}
    }
     - custom_db
        request_period_in_seconds: <request_period_in_seconds>
        sql_query: "<sql_query>"
        database: "<database>"
        schema: "<schema>"
        role: "<role>"
        override_devo_tag: "<override_tag_value>"

}
}

Replace Please replace the placeholders with your required real world values following the description table below:

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-snowflake_if-docker-image-1.4.1.tgz

031d6e40a17b99c7936d7fcde650ead6c90d5e1250002b2b72f23bf8cfb29f5e

Use the following command to add the Docker image to the system:

gunzip -c <image_file>-<version>.tgz | docker load

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

IMAGE_VERSION=<version> docker-compose up -d

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

1. In the Collector Server GUI, access the domain in which you want this instance to be created

2. Click Add Collector and find the one you wish to add.

3. In the Version field, select the latest value.

4. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).

5. In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.

6. In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

{
  "global_overrides": {
    "debug": false
  },
  "inputs": {
    "snowflake": {
      "id": "<input_id>",
      "enabled": <input_status>,
      "warehouse": "<warehouse>",
      "pager": <pager>,
      "credentials": {
        "username": "<username>",
        "password": "<password>",
        "account_identifier": "<account_identifier>"
      },
      "services": {
        "access_history": {
          "historic_date_utc": "<historic_date_utc>",
          "request_period_in_seconds": <request_period_in_seconds>,
          "role": "<role>"
        },
        "login_history": {
          "historic_date_utc": "<historic_date_utc>",
          "request_period_in_seconds": <request_period_in_seconds>,
          "role": "<role>"
        },
        "sessions": {
          "historic_date_utc": "<historic_date_utc>",
          "request_period_in_seconds": <request_period_in_seconds>,
          "role": "<role>"
        },
        "custom_sql": {
          "types": ["custom_db"],
          "request_period_in_seconds": <request_period_in_seconds>,
          "sql_query": "<sql_query>",
          "database": "<database>",
          "schema": "<schema>",
          "role": "<role>",
          "override_devo_tag": "<override_tag_value>"
        }
      }
    }
  }
}

Please replace the placeholders with real world values following the description table below:

Parameter

Data type

Type

Value range / Format

Details

input_id

str

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

input_status

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

warehouse

str

Mandatory

Minimum length: 1

This is the field to set the warehouse used to compute and generate data.

pager

int

Mandatory

Minimum value: 1

Use this param to define the number of results returned for each query performed.

username

str

Mandatory

Minimum length: 1

This is the field to set the username.

password

str

Mandatory

Minimum length: 1

This is the field to set the password.

account_identifier

str

Mandatory

Minimum length: 1

This is the field to set the Account Identifier.

request_period_in_seconds

int

Optional

Minimum length: 1

Period in seconds used between each data pulling, this value will overwrite the default value (300 seconds)

historic_date_utc

str

Mandatory

UTC with format: YYYY-mm-ddTHH:MM:SS.sssZ

This configuration allows you to set a custom date as the beginning of the period to download. This allows downloading historical data (1 month back for example) before downloading new events.

sql_query

str

Mandatory, if using custom_sql

Minimum length: 1

If using custom_sql service, this is the field to set the custom query.

database

str

Mandatory, if using custom_sql

Minimum length: 1

If using custom_sql service, this is the field to set the database.

schema

str

Mandatory, if using custom_sql

Minimum length: 1

If using custom_sql service, this is the field to set the schema.

role

str

Mandatory, if using custom_sql

Minimum length: 1

If using custom_sql service, this is the field to set the role. If using any other service it will default to ACCOUNTADMIN if not specified.

override_devo_tag

str

Optional, available only for custom_sql

Minimum length: 1

This would look like my.app.<type>.<subtype>

Image Removed

Image Added

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml 

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Image Added

Editing the config.yaml file

globals:
  debug: false
  id: "<collector_id>"
  name: "<collector_name>"
  persistence:
    type: filesystem
    config:
      directory_name: state
outputs:
  devo_us_1:
    type: devo_platform
    config:
      address: "<devo_address>"
      port: 443
      type: SSL
      chain: "<chain_filename>"
      cert: "<cert_filename>"
      key: "<key_filename>"
inputs:
  snowflake:
    id: "<input_id>"
    enabled: <input_status>
    warehouse: "<warehouse>"
    pager: <pager>
    credentials:
      username: "<username>"
      password: "<password>"
      account_identifier: "<account_identifier>"
    services:
      access_history:
        historic_date_utc: "<historic_date_utc>"
        request_period_in_seconds: <request_period_in_seconds>
        role: "<role>"
      login_history:
        historic_date_utc: "<historic_date_utc>"
        request_period_in_seconds: <request_period_in_seconds>
        role: "<role>"
      sessions:
        historic_date_utc: "<historic_date_utc>"
        request_period_in_seconds: <request_period_in_seconds>
        role: "<role>"
      custom_sql:
        types:
          - custom_db
        request_period_in_seconds: <request_period_in_seconds>
        sql_query: "<sql_query>"
        database: "<database>"
        schema: "<schema>"
        role: "<role>"
        override_devo_tag: "<override_tag_value>"

Replace the placeholders with your required values following the description table below:

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Use the following command to add the Docker image to the system:

gunzip -c <image_file>-<version>.tgz | docker load

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

IMAGE_VERSION=<version> docker-compose up -d

Collector services detail

...