Proofpoint on Demand collector

[ 1 Overview ] [ 2 Devo collector features ] [ 3 Data sources ] [ 4 Vendor setup ] [ 5 Run the collector ] [ 6 API limits and duplicates ] [ 7 Change log ]

Overview

Proofpoint on Demand API is designed for securing and managing email communications within your organization. The API provides access to Proofpoint's email protection services, including threat analysis, filtering, and reporting in real time.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`not allowed`
Running environments	`collector server` `on-premise`

Data sources

Source	Description	Devo table

Source	Description	Devo table
Message	Application API provides operations to manage applications and/or assignments to users or groups for your organization.	`mail.proofpoint.pod.message`
Mail Log	Allows for the retrieval and analysis of detailed mail logs, giving insights into email traffic patterns, delivery status, and threat detection activities.	`mail.proofpoint.pod.maillog`

For more information on how the events are parsed, visit our page.

Vendor setup

Getting Proofpoint On Demand (PoD) credentials

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

API limits and duplicates

Number of connections

The number of connections that can be performed with one credential set is limited. The credentials cannot pull data from more sources than the defined ones. If the access token is being used by another session, the API will return a 409 error: Exceeded maximum number of sessions per token.

The collector logs will show this error:

2024-08-09T10:48:10.163   ERROR InputProcess::ProofpointOnDemandWSSPuller(proofpoint_on_demand,12345,message,predefined) -> Handshake status 409 Conflict -+-+- {'date': 'Fri, 09 Aug 2024 08:48:10 GMT', 'content-type': 'text/plain;charset=iso-8859-1', 'content-length': '47'} -+-+- b'Exceeded maximum number of sessions per token\r\n' - goodbye

Duplicated events

It was observed that the API sometimes sends duplicate events. The collector can filter out duplicate events within an hour.

Change log

Release	Released on	Release type	Recommendations

Release	Released on	Release type	Recommendations
`v1.2.2`	Oct 7, 2024	improvement	`Recommended version`
`v1.2.2`	Improvements Upgrade DCSDK from `v1.12.2` to `v1.12.4` Improve Controlled stop when InputProcess is killed Change internal queue management for protecting against OOMK Extracted ModuleThread structure from PullerAbstract Improve Controlled stop when both processes fails to instatiate Fixed error related a ValueError exception not well controlled. Fixed error related with loss of some values in internal messages (collector_name, collector_id and job_id)
`v1.2.1`	Aug 8, 2024	BUG FIXING	`Update`
`v1.2.1`	Bug fixing Downgrade DCSDK from `v1.12.3` to `v1.12.2`
`v1.2.0`	Aug 7, 2024	improvement BUG FIXING	`Update`
`v1.2.0`	Improvements Upgrade DCSDK from `v1.12.2` to `v1.12.3` Bug fixing Reduce in-memory cached data to avoid memory issues
`v1.1.0`	Aug 6, 2024	NEW FEATURE improvement BUG FIXING	`Update`
`v1.1.0`	New features New parameters `override_tag_base` and `override_url_base` for `config.yaml` Parametrize `timestamp_field` and `datetime_format` to `collector_definitions.yaml` Reduce memory usage by changing `time_window_hours` to 1 Send messages and flush the `ProcessingLayer`'s cache immediately on connection close Optimize `ProcessingLayer`'s performance Detection when `start_time` has been changed to use it instead of persisted data (creates persistence v2) Persistence data structure automatic migration from v1 to v2 Adapt unit tests to new functionalities Mock web-socket server with Proofpoint POD API specifics for integration tests without credentials: Rounding down `sinceTime` param to the nearest hour Some events coming unsorted Bug fixing High CPU usage caused by a wait mechanism not working correctly Reduce persisted data size, causing memory issues (INT-2562, INT-2489) Improved duplicate filtering (2509) Improvements Upgrade DCSDK to v1.12.2 from v1.12.1 Upgrade DevoSDK dependency to version v5.4.0
`v1.0.1`	Apr 8, 2024	BUG FIXING	`Upgrade`
`v1.0.1`	Bug fixing: Added reset mechanism for stats counters to avoid growing them indefinitely
`v1.0.0`	Apr 5, 2024	NEW FEATURE	`Initial version`
`v1.0.0`