Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Proofpoint on Demand API is designed for securing and managing email communications within your organization. The API provides access to Proofpoint's email protection services, including threat analysis, filtering, and reporting in real time.

Devo collector features

Feature

Details

Allow parallel downloading (multipod)

not allowed

Running environments

  • collector server

  • on-premise

Data sources

Source

Description

Devo table

Message

Application API provides operations to manage applications and/or assignments to users or groups for your organization.

mail.proofpoint.pod.message

Mail Log

Allows for the retrieval and analysis of detailed mail logs, giving insights into email traffic patterns, delivery status, and threat detection activities.

mail.proofpoint.pod.maillog

...

Use the following command to add the Docker image to the system:

Rw ui tabs macro
Rw tab
titleOn-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block
<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml 
Note

Replace <product_name> with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

image-20240410-082550.png
Note

Replace <product_name> with the proper value.

Editing the config.yaml file

Code Block
globals:
  debug: false
  id: not_used
  name: proofpoint-on-demand
  persistence:
    type: filesystem
    config:
      directory_name: state

outputs:
  devo_1:
    type: devo_platform
    config:
      address: collector-us.devo.io
      port: 443
      type: SSL
      chain: chain.crt
      cert: <devo_domain>.crt
      key: <devo_domain>.key

inputs:
  proofpoint_on_demand:
    id: <unique_id>
    enabled: true
    credentials:
      cluster_id: <cluster_id_value>
      api_key: <api_key_value>
    environment: prod
    services:
      message:
        request_period_in_seconds: <request_period>period_in_seconds>
        start_time: <start_time_time>utc>
        maillog:override_tag_base: <override_tag_base>
        requestoverride_periodurl_in_secondsbase: <request_period><override_url_base>
      maillog:
        start_time: <start_time>
Info
All defined service entities will be executed by the collector. If you do
request_period_in_seconds: <request_period_in_seconds>
        start_time: <start_time_utc>
        override_tag_base: <override_tag_base>
        override_url_base: <override_url_base>
Info

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Replace the placeholders with your required values following the description table below:

Parameter

Data type

Type

Value range

Details

collector_id

str

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this collector.

collector_name

str

Mandatory

Minimum length: 1
Maximum length: 10

Use this param to give a valid name to this collector.

multiprocessing_mode

bool

Mandatory

false / true

If the value is true, the collector will run using a multiprocessing architecture. If the value is false, the collector will use only one CPU.

devo_address

str

Mandatory

collector-us.devo.io
collector-eu.devo.io

Use this param to identify the Devo Cloud where the events will be sent.

chain_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the chain.cert  file downloaded from your Devo domain. Usually this file's name is: chain.crt

cert_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.cert downloaded from your Devo domain.

key_filename

str

Mandatory

Minimum length: 4
Maximum length: 20

Use this param to identify the file.key downloaded from your Devo domain.

input_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

api_token

str

Mandatory

Any

API token to authenticate to the service.

period_in_seconds

int

Mandatory

Minimum length: 1

Recommended value: 60

This parameter allows you to customize this behavior for each service. As this collector uses websockets, this is the period elapsed between reconnections.

start_time

int

Mandatory

Following RFC 3339: %Y-%m-%dT%H:%M:%S.%f%z

Example: 2024-04-04T05:50:00.000-0500

Initial time period used when fetching data from the endpoint.

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-proofpoint_pod-docker-image-1.0.1

3db307ee5b2bd5b0a72e125c55f5009be74fe6ce21fb1bd6b36d0b55886df2bf

Code Block
gunzip -c <image_file>-<version>.tgz | docker load
Note

Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Blockdocker run --name collector-<product_name> --volume $PWD/certs:/devo-collector/certs --volume $PWD/config:/devo-collector/config

override_tag_base

str

Optional

str.str.str.str

Example: mail.proofpoint.pod.maillog

This parameter allows to override the destination table in Devo.

override_url_base

str

Optional

Valid connection URL

This parameter allows to change the connection URL to the websocket

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-proofpoint_pod-docker-image-1.1.0

473b77663e550fed0d807f0e002647936a97d9872c4931f73fcbe28e9498de67

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz | docker load
Note

Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block
docker run 
--name collector-<product_name> 
--volume $PWD/statecerts:/devo-collector/statecerts 
--env CONFIG_FILE=config.yaml 
volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>
Note

Replace <product_name>, <image_name> and <version> with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block
version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note

Replace <product_name>, <image_name> and <version> with the proper values.

Rw tab
titleCloud collector

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

  1. In the Collector Server GUI, access the domain in which you want this instance to be created

  2. Click Add Collector and find the one you wish to add.

  3. In the Version field, select the latest value.

  4. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).

  5. In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.

  6. In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

Code Block
{
  "global_overrides": {
    "debug": false
  },
  "inputs": {
    "proofpoint_on_demand": {
      "id": "<short_unique_id>",
      "enabled": true,
      "credentials": {
        "cluster_id": "<cluster_id>",
        "api_key": "<api_key>"
      },
      "environment": "prod",
      "services": {
        "message": {
          "request_period_in_seconds": "<request_period>_period_in_seconds>",
          "start_time": "<start_time_time>"
   utc>",
    },         "maillog"override_tag_base": {"<override_tag_base>",
          "request_period_in_seconds": <request_period>,
          "start_time": "<start_time>"
        }
      }
    }
  }
}

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Please replace the placeholders with real world values following the description table below:

Parameter

Data type

Type

Value range / Format

Details

input_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

enabled

bool

Mandatory

false / true

If the value is true, the input definition will be executed. If the value is false, the service will be ignored.

cluster_id

str

Mandatory

Any

Cluster id to get data from.

api_key

str

Mandatory

Any

API key to authenticate to the service.

request_period_in_seconds

int

Mandatory

Minimum length: 1

This parameter allows you to customize this behavior for each service.

start_time

int

Mandatory

Following RFC 3339: %Y-%m-%dT%H:%M:%S.%f%z

Example: 2024-04-04T05:50:00.000-0500

Initial time period used when fetching data from the endpoint.

Note

Due to the large amount of data produced by this service, using this parameter is discouraged except in special cases.

This parameter can be left blank, removed or commented.

Change log

...

Release

...

Released on

...

Release type

...

Details

...

override_url_base": "<override_url_base>"
        },
        "maillog": {
          "request_period_in_seconds": "<request_period_in_seconds>",
          "start_time": "<start_time_utc>",
          "override_tag_base": "<override_tag_base>",
          "override_url_base": "<override_url_base>"
        }
      }
    }
  }
}

All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.

Please replace the placeholders with real world values following the description table below:

Parameter

Data type

Type

Value range / Format

Details

input_id

int

Mandatory

Minimum length: 1
Maximum length: 5

Use this param to give an unique id to this input service.

enabled

bool

Mandatory

false / true

If the value is true, the input definition will be executed. If the value is false, the service will be ignored.

cluster_id

str

Mandatory

Any

Cluster id to get data from.

api_key

str

Mandatory

Any

API key to authenticate to the service.

request_period_in_seconds

int

Mandatory

Minimum length: 1

This parameter allows you to customize this behavior for each service.

start_time

int

Mandatory

Following RFC 3339: %Y-%m-%dT%H:%M:%S.%f%z

Example: 2024-04-04T05:50:00.000-0500

Initial time period used when fetching data from the endpoint.

Note

Due to the large amount of data produced by this service, using this parameter is discouraged except in special cases.

This parameter can be left blank, removed or commented.

override_tag_base

str

Optional

str.str.str.str

Example: mail.proofpoint.pod.maillog

This parameter allows to override the destination table in Devo.

override_url_base

str

Optional

Valid connection URL

This parameter allows to change the connection URL to the websocket

Change log

Release

Released on

Release type

Details

Recommendations

v1.1.0

Status
colourGreen
titleNEW FEATURE
Status
colourRed
titleBUG FIXING

New features

  • New parameters override_tag_base and override_url_base for config.yaml

  • Parametrize timestamp_field and datetime_format to collector_definitions.yaml

  • Reduce memory usage by changing time_window_hours to 1

  • Send messages and flush the ProcessingLayer's cache immediately on connection close

  • Optimize ProcessingLayer's performance

  • Detection when start_time has been changed to use it instead of persisted data (creates persistence v2)

  • Persistence data structure automatic migration from v1 to v2

  • Adapt unit tests to new functionalities

  • Mock web-socket server with Proofpoint POD API specifics for integration tests without credentials:

    • Rounding down sinceTime param to the nearest hour

    • Some events coming unsorted

Bug fixing

  • High CPU usage caused by a wait mechanism not working correctly

  • Reduce persisted data size, causing memory issues (INT-2562, INT-2489)

  • Improved duplicate filtering (2509)

Improvements

  • Upgrade DCSDK to v1.12.2 from v1.12.1

    • Upgrade DevoSDK dependency to version v5.4.0

Recommended version

v1.0.1

Status
colourRed
titleBUG FIXING

Bug fixing:

  • Added reset mechanism for stats counters to avoid growing them indefinitely

Recommended version

Upgrade

v1.0.0

Status
colourGreen
titleNEW FEATURE

New features:

  • Used DCSDK 1.11.1

  • Based on websocket-client

  • Created services:

    • message

    • maillog

Initial version