Proofpoint Tap collector

[ 1 Overview ] [ 2 Devo collector features ] [ 3 Data sources ] [ 4 Flattening preprocessing ] [ 5 Accepted authentication methods ] [ 6 Minimum configuration required for basic pulling ] [ 7 Run the collector ] [ 8 Collector services detail ] [ 9 Collector operations ] [ 10 Change log ]

Overview

Proofpoint Targeted Attack Protection (TAP) helps you stay ahead of attackers with an innovative approach that detects, analyzes and blocks advanced threats.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`not allowed`
Running environments	`collector server` `on-premise`
Populated Devo events	`table`
Flattening preprocessing	`no`

Data sources

Data source	Description	API endpoint	Collector service name	Devo table	Available from release

Data source	Description	API endpoint	Collector service name	Devo table	Available from release
`Blocked clicks`	Fetch events for clicks to malicious URLs blocked in the specified time period	`/v2/siem/clicks/blocked`	`clicksBlocked`	`mail.proofpoint.tapsiem_v2.clicksblocked`	`v2.1.1`
`Permitted clicks`	Fetch events for clicks to malicious URLs permitted in the specified time period	`/v2/siem/clicks/permitted`	`clicksPermitted`	`mail.proofpoint.tapsiem_v2.clickspermitted`	`v2.1.1`
`Blocked messages`	Fetch events for messages blocked in the specified time period that contained a known threat.	`/v2/siem/messages/blocked`	`messagesBlocked`	`mail.proofpoint.tapsiem_v2.messagesblocked`	`v2.1.1`
`Delivered message`	Fetch events for messages delivered in the specified time period which contained a known threat.	`/v2/siem/messages/delivered`	`messagesDelivered`	`mail.proofpoint.tapsiem_v2.messagesdelivered`	`v2.1.1`

For more information on how the events are parsed, visit our page.

Flattening preprocessing

Data source	Collector service	Optional	Flattening details

Data source	Collector service	Optional	Flattening details
`Blocked messages`	`messagesBlocked`	`yes`	not required
`Delivered message`	`messagesDelivered`	`yes`	not required

Accepted authentication methods

Authentication method	username	password

Authentication method	username	password
`credentials`	REQUIRED	REQUIRED

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details

Setting	Details
`username`	The username for proofpoint Tap
`password`	The password(credential) for proofpoint
`start_time_in_utc_format`	Start Time which is not more than 7 days into the past

See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Messages blocked

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2023-12-26T09:05:33.934    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:05:33.935    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:05:33.935    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:05:33.935    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:05:33.935    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.935    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935    INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:05:33.936    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.939    INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:05:33.940    INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:05:33.940    INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:05:33.941    INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:05:33.941    INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:05:33.941    INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:05:33.941    INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:05:33.941    INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:05:33.941    INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f75040e44c0>
2023-12-26T09:05:33.941    INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) Finalizing the execution of init_variables()
2023-12-26T09:05:33.942    INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:05:33.942    INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:05:33.942    INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Starting thread
2023-12-26T09:05:33.942 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:05:33.943    INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread
2023-12-26T09:05:33.943 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Waiting until setup will be executed
2023-12-26T09:05:33.947    INFO OutputProcess::MainThread -> [GC] global: 28.2% -> 28.2%, process: RSS(40.77MiB -> 41.64MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:05:33.954    INFO InputProcess::MainThread -> [GC] global: 28.2% -> 28.2%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:05:34.739    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140140610316704"
2023-12-26T09:05:36.819    INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Setup for module <ProofPointPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

023-12-26T09:05:36.953    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Pull Started
2023-12-26T09:05:36.956    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-10 11:00:00+00:00 TO: 2023-12-10 12:00:00+00:00
2023-12-26T09:05:36.956 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Start_Time_in_utc must be at most 7.00d into the past, Changing the time startTime to be in the specified time
2023-12-26T09:05:39.320    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:39.321    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 04:35:36.946633+00:00 TO: 2023-12-19 05:35:36.946633+00:00
2023-12-26T09:05:41.780    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:41.781    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 05:35:36.946633+00:00 TO: 2023-12-19 06:35:36.946633+00:00
2023-12-26T09:05:44.135    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:44.136    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 06:35:36.946633+00:00 TO: 2023-12-19 07:35:36.946633+00:00
2023-12-26T09:05:46.495    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 4; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 0.105.

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2023-12-26T09:05:46.495    INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 4; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 0.105.

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

Error type	Error ID	Error message	Cause	Solution

Error type	Error ID	Error message	Cause	Solution
SetupError	100	`HTTP Error occurred while retrieving events from Proof point server`	username and password is not correct	Make sure that credentials are correct.
SetupError	101	`Error occurred while retrieving events from ProofPoint server`	start_time_in_utc is in future or not in proper format	Make sure the start time is not in future and not in proper format
PullError	300	`HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}`	This error happens when the collector tries to fetch the data from API.	In this error you will find the HTTP error code as well as the summary and details.
	301	`Some Error occurred while retrieving events from ProofPoint server : {Exception}`	Some exception occured while making the API request.	Reach out to the developer with the exact error message.

Messages delivered

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Clicks permitted

Clicks blocked

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Change log

Release	Released on	Release type	Details	Recommendations

Release	Released on	Release type	Details	Recommendations
`v3.0.0`	Aug 26, 2024	IMPROVEMENTSFeatures	Improvements Upgraded the DCSDK from 1.11.0 to 1.12.4: Updated the base Docker Image to 1.3.0 Features Added 4 new enpdoints Threats Campaigns People vap top-clickers	`Recommended version`
`v2.2.1`	Feb 23, 2024	IMPROVEMENTS	Improvements Upgraded the DCSDK from 1.10.3 to 1.11.0: Updated DevoSDK to v5.1.10 Fix for SyslogSender related to UTF-8 Enhance of troubleshooting. Trace Standardization, Some traces has been introduced. Introduced a machanism to detect "Out of Memory killer" situation.	`Update`
`v2.2.0`	Jan 16, 2024	BUG IMPROVEMENTS	Bug Fixed the issue where the collector is unable to fetch data after a certain time period Improvements Upgraded the DCSDK docker base image version to 1.1.0	`Update`
`v2.1.1`	Dec 21, 2023	FIRST RELEASE	Released the first version of the Proofpoint TAP collector.	`Update`