Fetch events for clicks to malicious URLs blocked in the specified time period
/v2/siem/clicks/blocked
clicksBlocked
mail.proofpoint.tapsiem_v2.clicksblocked
v2.1.1
Permitted clicks
Fetch events for clicks to malicious URLs permitted in the specified time period
/v2/siem/clicks/permitted
clicksPermitted
mail.proofpoint.tapsiem_v2.clickspermitted
v2.1.1
Blocked messages
Fetch events for messages blocked in the specified time period that contained a known threat.
/v2/siem/messages/blocked
messagesBlocked
mail.proofpoint.tapsiem_v2.messagesblocked
v2.1.1
Delivered message
Fetch events for messages delivered in the specified time period which contained a known threat.
/v2/siem/messages/delivered
messagesDelivered
mail.proofpoint.tapsiem_v2.messagesdelivered
v2.1.1
For more information on how the events are parsed, visit our page.
Flattening preprocessing
...
Data source
...
Collector service
...
Optional
...
Flattening details
...
Blocked messages
...
messagesBlocked
...
yes
...
not required
...
Delivered message
...
messagesDelivered
...
yes
...
not required
Accepted authentication methods
...
Authentication method
...
username
...
password
...
credentials
...
Status
colour
Green
title
REQUIRED
...
Status
colour
Green
title
REQUIRED
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
Info
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
...
Setting
...
Details
...
username
...
The username for proofpoint Tap
...
password
...
The password(credential) for proofpoint
...
start_time_in_utc_format
...
Start Time which is not more than 7 days into the past
Info
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
...
Rw tab
title
On-premise collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.
Replace the placeholders with your required values following the description table below:
...
Parameter
...
Data type
...
Type
...
Value range / Format
...
Details
...
<short_unique_id>
...
int
...
Mandatory
...
Minimum Lenght 5
...
Use this param to give a unique id to this input service.
Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
...
<input_status>
...
bool
...
Mandatory
...
false / true
...
Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.
...
<username_value>
...
str
...
Mandatory
...
Minimum Length 1
...
Provide the value of username used for authentication
...
<password_value>
...
str
...
Mandatory
...
Minimum Length 1
...
Provide the value of password used for authentication
...
<start_time_in_utc_format>
...
str
...
Mandatory
...
Minimum Length 1
%Y-%m-%dT%H:%M:%SZ
...
This configuration allows you to set a custom date as the beginning of the period to download. This allows the events but it can’t be more than 7 days in the past.
...
request_period_in_seconds
...
int
...
Optional
...
Minimum Length 1
...
This configuration allows us to set the “seconds” after which next pull cycle to be made. Allowed limit of number of api requests is 1800 per day so we have kept 240 seconds as a default value.
...
<override_tag_value>
...
str
...
Optional
...
A devo tag
...
This parameter allows to define a custom devo tag.
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Replace <product_name>, <image_name> and <version> with the proper values.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:
Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note
Replace <product_name>, <image_name> and <version> with the proper values.
Rw tab
title
Cloud collector
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
...
Threats
The Threats API allows administrators to pull detailed attributes about individual threats observed in their environment. It can be used to retrieve more intelligence for threats identified in the SIEM or Campaign API responses
siem/all then -> v2/threat/summary/
threats
mail.proofpoint.tap_threats.events
Threats if Forensics is Enabled
If Forensics is enabled the events are flattened into the table
v2/threat/summary/ then -> v2/threat/summary/&includecampaignforensics=true
threats
mail.proofpoint.tap_threats.events
Campaigns
The Campaign API allows administrators to pull campaign IDs in a timeframe and specific details about campaigns, including: their description; the actor, malware family, and techniques associated with the campaign; and the threat variants which have been associated with the campaign
v2/campaign/
campaigns
mail.proofpoint.tap_campaigns.events
Campaigns if Forensics is Enabled
If Forensics is enabled the events are flattened into the table
v2/forensics?campaignId=
campaigns
mail.proofpoint.tap_campaigns.events
People Top Clicks
The People API allows administrators to identify which users in their organizations were most attacked or are the top clickers during a specified period. Fetch the identities and attack index of the top clickers within your organization for a given period. Top clickers are the users who have demonstrated a tendency to click on malicious URLs, regardless of whether the clicks were blocked or not. Knowing who are more susceptible to threats is useful for proactive security approaches such as security training assignments.
v2/people/top-clickers
people_topclicks
mail.proofpoint.tap_people.top_clicks
People VAP
The People API allows administrators to identify which users in their organizations were most attacked or are the top clickers during a specified period. Fetch the identities and attack index breakdown of Very Attacked People within your organization for a given period.
v2/people/vap
people_vap
mail.proofpoint.tap_people.vap
For more information on how the events are parsed, visit our page.
Flattening preprocessing
Data source
Collector service
Optional
Flattening details
Blocked messages
messagesBlocked
yes
not required
Delivered message
messagesDelivered
yes
not required
Accepted authentication methods
Authentication method
username
password
credentials
Status
colour
Green
title
REQUIRED
Status
colour
Green
title
REQUIRED
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
Info
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting
Details
username
The username for proofpoint Tap
password
The password(credential) for proofpoint
start_time_in_utc_format
Start Time which is not more than 7 days into the past
Info
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.
API Limits, issues
for services clicksBlocked, clicksPermitted, messageBlocked, messageDelivered
ClicksBlocked (/v2/siem/clicks/blocked) , messageBlocked (/v2/siem/messages/blocked), messageDelivered (/v2/siem/messages/delivered) can collectively make 1800 requests per day as per api documentation
clicksPermitted (/v2/siem/clicks/permitted) can make 1800 reuqests per day as per api documentation.
To simplify the code and handle all the four services, code has been written in a way that each of the four service can make 220 request per day.
We have put a limit on the api call,which will block the api requests after 220 requests for a service.
If the collector took, let’s say, 15 hours to make 220 requests, the collector will block the api call for 9 hours (9 + 15 = 24 hours) and it will resume making api requests after 9 hours. For these 9 hours no ingestion will occur for that service.
Issues:
Let’s say all the services make 1000 request in 15 hours and then due to some issue collector restarts. Now, after the restart collector is going to assume that service still make a 1800 request , ignoring the 1000 requests collector made before the restart. This will lead to surpass the 1800 requets per day api limit causing a 429 error.
for services threats, campaigns, people_topclicks, people_vap
threats service has no api limit
people_topclicks (v2/people/top-clickers) and people_vap (v2/people/vap) services can make only 50 requests in 1 day as per the api documentation
campaigns (v2/campaign/ids) service can only make 50 request in 1 day as per the api documentation
We have put a limit on the api call,which will block the api requests after 50 requests for a service
If the collector took , let’s say, 10 hours to make 50 request, the collector will block the api call for 14 hours (10 + 14 = 24 hours) and it will resume making api requests after 14 hours. For these 14 hours no ingestion will occur for that service
Issues:
Let’s say a service make 30 request in 3 hours and then due to some issue collector restarts. Now, after the restart collector is going to assume that service still make a 50 request , ignoring the 30 requests collector made before the restart. This will lead to surpass the 50 requets per day api limit causing a 429 error for that service.
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Rw ui tabs macro
Rw tab
title
Cloud collector
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration. Editing the json file
Use this param to give a unique id to this input service.
Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
<input_status>
bool
Mandatory
false / true
Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.
<username_value>
str
Mandatory
Minimum Length 1
Provide the value of username used for authentication
<password_value>
str
Mandatory
Minimum Length 1
Provide the value of password used for authentication
<start_time_in_utc_format>
str
Mandatory
Minimum Length 1
%Y-%m-%dT%H:%M:%SZ
This configuration allows you to set a custom date as the beginning of the period to download. This allows the events but it can’t be more than 7 days in the past.
request_period_in_seconds
int
Optional
Minimum Length 1
This configuration allows us to set the “seconds” after which next pull cycle to be made. Allowed limit of number of api requests is 1800 per day so we have kept 600 seconds as a default value.
<override_tag_value>
str
Optional
A devo tag
This parameter allows to define a custom devo tag.
Rw tab
title
On-premise collector
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object.
Replace the placeholders with your required values following the description table below:
Parameter
Data type
Type
Value range / Format
Details
<short_unique_id>
int
Mandatory
Minimum Lenght 5
Use this param to give a unique id to this input service.
Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.
<input_status>
bool
Mandatory
false / true
Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.
<username_value>
str
Mandatory
Minimum Length 1
Provide the value of username used for authentication
<password_value>
str
Mandatory
Minimum Length 1
Provide the value of password used for authentication
<start_time_in_utc_format>
str
Mandatory
Minimum Length 1
%Y-%m-%dT%H:%M:%SZ
This configuration allows you to set a custom date as the beginning of the period to download. This allows the events but it can’t be more than 7 days in the past.
request_period_in_seconds
int
Optional
Minimum Length 1
This configuration allows us to set the “seconds” after which next pull cycle to be made. Allowed limit of number of api requests is 1800 per day so we have kept 600 seconds as a default value.
<override_tag_value>
str
Optional
A devo tag
This parameter allows to define a custom devo tag.
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file> and <version> with a proper value.
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Replace <product_name>, <image_name> and <version> with the proper values.
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:
Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note
Replace <product_name>, <image_name> and <version> with the proper values.
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Messages blocked
Info
No. of request this service can make in a day is 220.
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
2023-12-26T09:05:33.934 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:05:33.935 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:05:33.936 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.939 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f75040e44c0>
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) Finalizing the execution of init_variables()
2023-12-26T09:05:33.942 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:05:33.942 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:05:33.942 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Starting thread
2023-12-26T09:05:33.942 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:05:33.943 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread
2023-12-26T09:05:33.943 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Waiting until setup will be executed
2023-12-26T09:05:33.947 INFO OutputProcess::MainThread -> [GC] global: 28.2% -> 28.2%, process: RSS(40.77MiB -> 41.64MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:05:33.954 INFO InputProcess::MainThread -> [GC] global: 28.2% -> 28.2%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:05:34.739 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140140610316704"
2023-12-26T09:05:36.819 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
023-12-26T09:05:36.953 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Pull Started
2023-12-26T09:05:36.956 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-10 11:00:00+00:00 TO: 2023-12-10 12:00:00+00:00
2023-12-26T09:05:36.956 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Start_Time_in_utc must be at most 7.00d into the past, Changing the time startTime to be in the specified time
2023-12-26T09:05:39.320 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:39.321 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 04:35:36.946633+00:00 TO: 2023-12-19 05:35:36.946633+00:00
2023-12-26T09:05:41.780 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:41.781 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 05:35:36.946633+00:00 TO: 2023-12-19 06:35:36.946633+00:00
2023-12-26T09:05:44.135 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:44.136 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 06:35:36.946633+00:00 TO: 2023-12-19 07:35:36.946633+00:00
2023-12-26T09:05:46.495 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 4; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 0.105.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
2023-12-26T09:05:46.495 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 4; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 0.105.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Expand
title
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
Messages delivered
Info
No. of request this service can make in a day is 220.
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:01.713 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2023-12-26T09:15:04.224 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Pull Started
2023-12-26T09:15:04.227 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 03:00:00+00:00 TO: 2023-12-25 04:00:00+00:00
2023-12-26T09:15:12.140 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:15:12.140 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 04:00:00+00:00 TO: 2023-12-25 05:00:00+00:00
2023-12-26T09:15:13.985 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:15:13.986 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 05:00:00+00:00 TO: 2023-12-25 06:00:00+00:00
2023-12-26T09:15:16.077 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:15:16.077 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 06:00:00+00:00 TO: 2023-12-25 07:00:00+00:00
2023-12-26T09:15:18.287 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 4; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
2023-12-22T10:28:58.153 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 164; Number of events received: 14; Number of duplicated events filtered out: 0; Number of events generated and sent: 14; Average of events per second: 0.047.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Expand
title
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
Clicks permitted
Info
No. of request this service can make in a day is 220.
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
2023-12-26T08:57:18.936 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T08:57:18.957 WARNING InputProcess::MainThread -> A previous rate limiter with same "period_in_seconds" and "number_of requests" was already existing: "86400/1800"
2023-12-26T08:57:18.957 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) Starting the execution of init_variables()
2023-12-26T08:57:18.957 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T08:57:18.959 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T08:57:18.962 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T08:57:18.963 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T08:57:18.963 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T08:57:18.963 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.963 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T08:57:18.964 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T08:57:18.964 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T08:57:18.964 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f2d62685640>
2023-12-26T08:57:18.964 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.964 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) Finalizing the execution of init_variables()
2023-12-26T08:57:18.964 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.964 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread (execution_period=60s)
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Starting thread
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread
2023-12-26T08:57:18.965 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Waiting until setup will be executed
2023-12-26T08:57:18.965 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> The token/header/authentication has not been created yet
2023-12-26T08:57:18.978 INFO InputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.04MiB -> 40.04MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T08:57:18.979 INFO OutputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.52MiB -> 41.39MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T08:57:19.718 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139832919561120"
2023-12-26T08:57:22.226 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2023-12-26T08:57:22.974 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Pull Started
2023-12-26T08:57:22.980 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 13:48:14+00:00 TO: 2023-12-21 14:48:14+00:00
2023-12-26T08:57:25.954 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:25.955 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 14:48:14+00:00 TO: 2023-12-21 15:48:14+00:00
2023-12-26T08:57:27.948 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:27.949 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 15:48:14+00:00 TO: 2023-12-21 16:48:14+00:00
2023-12-26T08:57:29.784 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:29.785 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 16:48:14+00:00 TO: 2023-12-21 17:48:14+00:00
2023-12-26T08:57:31.515 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 4; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:31.516 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 17:48:14+00:00 TO: 2023-12-21 18:48:14+00:00
2023-12-26T08:57:33.312 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 5; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:33.313 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 18:48:14+00:00 TO: 2023-12-21 19:48:14+00:00
2023-12-26T08:57:42.836 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
2023-12-26T08:57:42.836 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Expand
title
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
Clicks blocked
Info
No. of request this service can make in a day is 220.
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) - Starting thread
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread (execution_period=60s)
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Starting thread
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> The token/header/authentication has not been created yet
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread (execution_period=60s)
2023-12-21T19:10:16.128 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Starting thread
2023-12-21T19:10:16.129 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread
2023-12-21T19:10:16.129 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> The token/header/authentication has not been created yet
2023-12-21T19:10:16.130 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.131 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-21T19:10:16.131 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-21T19:10:16.131 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-21T19:10:16.131 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-21T19:10:16.131 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.145 INFO InputProcess::MainThread -> [GC] global: 32.2% -> 32.2%, process: RSS(40.54MiB -> 40.54MiB), VMS(1.05GiB -> 1.05GiB)
2023-12-21T19:10:16.768 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140555598948768"
2023-12-21T19:10:18.447 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2023-12-22T10:24:02.739 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Pull Started
2023-12-22T10:24:02.741 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Time Window FROM: 2023-12-10 11:00:00+00:00 TO: 2023-12-10 12:00:00+00:00
2023-12-22T10:24:02.742 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Start_Time_in_utc must be at most 7.00d into the past, Changing the time startTime to be in the specified time
2023-12-22T10:24:04.567 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-22T10:24:04.568 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Time Window FROM: 2023-12-15 05:54:02.731674+00:00 TO: 2023-12-15 06:54:02.731674+00:00
2023-12-22T10:24:06.173 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-22T10:24:06.174 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Time Window FROM: 2023-12-15 06:54:02.731674+00:00 TO: 2023-12-15 07:54:02.731674+00:00
2023-12-22T10:24:07.839 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
2023-12-26T08:57:42.836 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Expand
title
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
Threats
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
2023-12-26T09:0515:3301.934181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:0515:3301.935204 INFO OutputProcessInputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
Validating service metadata
2023-12-26T09:0515:3301.935206 INFO OutputProcessInputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting threadValidating defined module definition
2023-12-26T09:0515:3301.935207 INFO OutputProcess::MainThread -> DevoSender(internalstandard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:0515:3301.935208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internalstandard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:0515:3301.935208 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1)MainThread -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookupstandard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
Starting thread
2023-12-26T09:0515:3301.935208 INFO OutputProcess::MainThread -> DevoSenderManagerDevoSender(internallookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935 INFO InputProcess::MainThread -> Validating defined module definition
,devo_sender_0) -> Starting thread
2023-12-26T09:0515:3301.936208 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1)MainThread -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internalDevoSenderManagerMonitor(lookup_senders,manager,devo_us_1) -> NothingStartingretrievedthreadfrom(everythe300persistence.seconds)
2023-12-26T09:0515:3301.936208 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0)MainThread -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.Starting thread
2023-12-26T09:0515:3301.936209 INFO OutputProcess::OutputLookupConsumerOutputStandardConsumer(lookupstandard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumerOutputStandardConsumer(lookupstandard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.939 INFO InputProcess::MainThread -> Validating common input config retrieved from the persistence.
2023-12-26T09:0515:3301.940209 INFO InputProcess::MainThread -> Validating servicecommon input config
2023-12-26T09:0515:3301.940209 INFO InputProcessOutputProcess::MainThreadDevoSenderManager(standard_senders,manager,devo_us_1) -> Running[EMERGENCYoverridingPERSISTENCErules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Validating the rate limiter config given by the userSYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:0515:3301.941209 INFO InputProcessOutputProcess::MainThreadDevoSenderManager(lookup_senders,manager,devo_us_1) -> <requests_limits> setting has not been defined. The generic settings will be used instead[EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:0515:3301.941209 INFO InputProcessOutputProcess::MainThread -> Adding raw config to the collector store
DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:0515:3301.941209 INFO InputProcess::MainThreadOutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> RunningNothing retrievedcustomfromvalidationtherulespersistence.
2023-12-26T09:0515:3301.941209 INFO InputProcessOutputProcess::MainThread -> Creating API client.DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:0515:3301.941209 INFO InputProcessOutputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f75040e44c0>DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:0515:3301.941209 INFO InputProcessOutputProcess::MainThreadDevoSenderManager(internal_senders,manager,devo_us_1) -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) Finalizing the execution of init_variables() [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:0515:3301.942209 INFO InputProcessOutputProcess::MainThreadOutputInternalConsumer(internal_senders_consumer_0) -> InputThread(proofpoint_tap,123456 [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) - Starting thread (execution_period=60s)> Nothing retrieved from the persistence.
2023-12-26T09:0515:3301.942210 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread (execution_period=60s)Validating service input config
2023-12-26T09:0515:3301.942210 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Starting thread
Running overriding rules
2023-12-26T09:0515:33.942 WARNING01.210 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined)MainThread -> The token/header/authentication has not been created yetValidating the rate limiter config given by the user
2023-12-26T09:0515:3301.943210 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread<requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:0515:33.943 WARNING01.211 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined)MainThread -> AddingWaitingrawuntilconfigsetuptowillthebecollectorexecutedstore
2023-12-26T09:0515:3301.947211 INFO OutputProcessInputProcess::MainThread -> [GC] global: 28.2% -> 28.2%, process: RSS(40.77MiB -> 41.64MiB), VMS(926.00MiB -> 926.00MiB)Running custom validation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:0515:3301.954211 INFO InputProcess::MainThread -> Created[GC]requestglobalclient: 28.2% -> 28.2%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)<agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:0515:3401.739211 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140140610316704"
2023-12-26T09:05:36.819 InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,threats,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,threats,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
023threats#predefined) -> Starting thread
2023-12-26T09:0515:3601.953212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlockedmessagesDelivered,predefined) ->PullStartingStartedthread
2023-12-26T09:0515:3601.956 INFO212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlockedmessagesDelivered,predefined) -> Time Window FROM: 2023-12-10 11:00:00+00:00 TO: 2023-12-10 12:00:00+00:00Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,threats#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:0515:36.956 WARNING01.225 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined)MainThread -> Start_Time_in_utc must be at most 7.00d into the past, Changing the time startTime to be in the specified time[GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:0515:3901.320713 INFO InputProcessOutputProcess::ProofPointPullerDevoSender(proofpointinternal_tap,123456,messagesBlocked,predefinedsenders,devo_sender_0) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:0515:3903.321324 INFO InputProcess::ProofPointPullerProofPointPullerSetup(proofpoint_tap_collector,123456,messagesBlocked,predefinedproofpoint_tap#123456,threats#predefined) -> TimeSetupWindowforFROM: 2023-12-19 04:35:36.946633+00:00 TO: 2023-12-19 05:35:36.946633+00:00
2023-12-26T09:05:41.780module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2025-03-05T16:28:00.760 INFO InputProcess::ProofPointPullerProofPointThreatsPuller(proofpoint_tap,123456,messagesBlocked,predefinedtap#1212090,threats#predefined) -> (Partial)PullStatistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:41.781Started
2025-03-05T16:28:00.762 INFO InputProcess::ProofPointPullerProofPointThreatsPuller(proofpoint_tap,123456,messagesBlocked,predefinedtap#1212090,threats#predefined) -> Time Window FROM: 20232024-1209-19260514:3500:36.94663300+00:00 TO: 20232024-1209-19260615:3500:36.94663300+00:00
20232025-1203-26T0905T16:0559:4432.135041 INFO InputProcess::ProofPointPullerProofPointThreatsPuller(proofpoint_tap,123456,messagesBlocked,predefinedtap#1212090,threats#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035617369461741170569801):Number of requests made: 31; Number of events received: 0193; Number of duplicated events filtered out: 0187; Number of events generated and sent: 06; Average of events per second: 02.000687.
20232025-1203-26T0905T16:0559:4432.136041 INFO InputProcess::ProofPointPullerProofPointThreatsPuller(proofpoint_tap,123456,messagesBlocked,predefinedtap#1212090,threats#predefined) -> Time WindowFROM: 2023-12-19 06:35:36.946633+00:00 TO: 2023-12-19 07:35:36.946633+00:00
2023-12-26T09:05:46.495 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035617369461741170569801):Number of requests made: 41; Number of events received: 1193; Number of duplicated events filtered out: 0187; Number of events generated and sent: 16; Average of events per second: 02.105687.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
20232025-1203-26T0905T16:0559:4632.495041 INFO InputProcess::ProofPointPullerProofPointThreatsPuller(proofpoint_tap,123456,messagesBlocked,predefinedtap#1212090,threats#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035617369461741170569801):Number of requests made: 41; Number of events received: 1193; Number of duplicated events filtered out: 0187; Number of events generated and sent: 16; Average of events per second: 02.105687.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
...
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
...
Campaigns
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcessInputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
Validating service metadata
2023-12-26T09:15:01.208206 INFO OutputProcessInputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting threadValidating defined module definition
2023-12-26T09:15:01.208207 INFO OutputProcess::MainThread -> DevoSender(lookupstandard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookupstandard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookupstandard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config thread
2023-12-26T09:15:01.209208 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1)MainThread -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standardDevoSender(lookup_senders,manager,devo_ussender_10) -> Nothing retrieved from the persistence.
Starting thread
2023-12-26T09:15:01.209208 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1)MainThread -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManagerDevoSenderManagerMonitor(lookup_senders,manager,devo_us_1) -> NothingStartingretrievedthreadfrom(everythe300persistence.seconds)
2023-12-26T09:15:01.209208 INFO OutputProcess::MainThread -> DevoSenderDevoSenderManager(internallookup_senders,manager,devo_senderus_01) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumerOutputStandardConsumer(lookupstandard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumerOutputStandardConsumer(lookupstandard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds) persistence.
2023-12-26T09:15:01.209 INFO OutputProcessInputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting threadValidating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(internalstandard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internalstandard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::OutputInternalConsumerDevoSenderManager(internallookup_senders,manager,devo_consumerus_01) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumerDevoSenderManager(internallookup_senders,manager,devo_consumerus_01) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.210209 INFO InputProcessOutputProcess::MainThread -> Validating service input configDevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.210209 INFO InputProcess::MainThreadOutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> NothingRunningretrieved fromoverridingtherulespersistence.
2023-12-26T09:15:01.210209 INFO InputProcessOutputProcess::MainThread -> Validating the rate limiter config given by the userDevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.210209 INFO InputProcessOutputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.211209 INFO InputProcessOutputProcess::MainThreadDevoSenderManager(internal_senders,manager,devo_us_1) -> Adding raw config to the collector store
[EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.211209 INFO InputProcessOutputProcess::MainThreadOutputInternalConsumer(internal_senders_consumer_0) -> Running[EMERGENCYcustomPERSISTENCEvalidation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API clientSYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.211210 INFO InputProcess::MainThread -> CreatedValidatingrequestserviceclient: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
input config
2023-12-26T09:15:01.211210 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Finalizing the execution of init_variables()
Running overriding rules
2023-12-26T09:15:01.212210 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s) Validating the rate limiter config given by the user
2023-12-26T09:15:01.212210 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)<requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:15:01.212211 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread Adding raw config to the collector store
2023-12-26T09:15:01.212211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting threadRunning custom validation rules
2023-12-26T09:15:01.212 WARNING211 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined)MainThread -> WaitingCreatinguntilAPIsetup will be executed
client.
2023-12-26T09:15:01.212 WARNING211 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined)MainThread -> The token/header/authentication has not been created yetCreated request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:15:01.221211 INFO OutputProcessInputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiBProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.225212 INFO InputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiBInputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.713212 INFO OutputProcessInputProcess::DevoSender(internal_senders,devo_sender_0MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) ->Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:0301.324 INFO212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
The token/header/authentication has not been created yet
2023-12-26T09:15:04.224 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Pull Started:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:0401.227225 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined)MainThread -> Time Window FROM: 2023-12-25 03:00:00+00:00 TO: 2023-12-25 04:00:00+00:00[GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:1201.140713 INFO InputProcessOutputProcess::ProofPointPullerDevoSender(proofpointinternal_tap,123456,messagesDelivered,predefinedsenders,devo_sender_0) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:1203.140324 INFO InputProcess::ProofPointPullerProofPointPullerSetup(proofpoint_tap_collector,123456,messagesDelivered,predefinedproofpoint_tap#123456,messagesDelivered#predefined) -> TimeSetupWindowforFROM: 2023-12-25 04:00:00+00:00 TO: 2023-12-25 05:00:00+00:00
2023-12-26T09:15:13.985module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2025-03-05T16:28:01.769 INFO InputProcess::ProofPointPullerProofPointCampaignsPuller(proofpoint_tap,123456,messagesDelivered,predefinedtap#1212090,campaigns#predefined) -> (Partial)PullStatistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:15:13.986Started
2025-03-05T16:28:01.770 INFO InputProcess::ProofPointPullerProofPointCampaignsPuller(proofpoint_tap,123456,messagesDelivered,predefinedtap#1212090,campaigns#predefined) -> Time Window FROM: 20232025-1202-25 0511:0030:0029+00:00 TO: 20232025-1202-25260611:0029:0029+00:00
20232025-1203-26T0905T16:1528:1607.077447 INFO InputProcess::ProofPointPullerProofPointCampaignsPuller(proofpoint_tap,123456,messagesDelivered,predefinedtap#1212090,campaigns#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035623042161741172281758):Number of requests made: 31; Number of events received: 0359; Number of duplicated events filtered out: 0359; Number of events generated and sent: 0; Average of events per second: 0.000.
20232025-1203-26T0905T16:15:16.077 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 06:00:00+00:00 TO: 2023-12-25 07:00:00+00:00
2023-12-26T09:15:18.28728:07.447 INFO InputProcess::ProofPointPullerProofPointCampaignsPuller(proofpoint_tap,123456,messagesDelivered,predefinedtap#1212090,campaigns#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035623042161741172281758):Number of requests made: 41; Number of events received: 0359; Number of duplicated events filtered out: 0359; Number of events generated and sent: 0; Average of events per second: 0.000.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
20232025-1203-22T1005T16:28:5807.153447 INFO InputProcess::ProofPointPullerProofPointCampaignsPuller(proofpoint_tap,123456,clicksBlocked,predefinedtap#1212090,campaigns#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17032208427311741172281758):Number of requests made: 1641; Number of events received: 14359; Number of duplicated events filtered out: 0359; Number of events generated and sent: 140; Average of events per second: 0.047000.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
...
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
...
People Topclicks
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T0826T09:5715:1801.936182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T0826T09:5715:18.957 WARNING01.182 INFO InputProcess::MainThread -> A previous rate limiter with same "period_in_seconds" and "number_of requests" was already existing: "86400/1800"
Process Started
2023-12-26T0826T09:5715:1801.957204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermittedmessagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T0826T09:5715:1801.957204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T08:57:18.959 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T0826T09:5715:1801.962206 INFO InputProcess::MainThread -> Validating commondefinedinputmoduleconfigdefinition
2023-12-26T0826T09:5715:1801.963207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T0826T09:5715:18.963 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T08:57:18.963 01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T0826T09:5715:1801.963208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T0826T09:5715:1801.963208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T0826T09:5715:1801.963208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T0826T09:5715:1801.963208 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T0826T09:5715:1801.963209 INFO OutputProcess::DevoSenderManagerOutputStandardConsumer(standard_senders,manager,devo_usconsumer_10) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManagerOutputStandardConsumer(standard_senders,manager,devo_usconsumer_10) -> Nothing retrieved from the persistence.
2023-12-26T0826T09:5715:18.963 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T08:57:18.964 01.209 INFO InputProcess::MainThread -> Validating thecommonrate limiterinput config given by theuser
2023-12-26T0826T09:5715:1801.964209 INFO OutputProcess::MainThread -> DevoSender(internalDevoSenderManager(standard_senders,manager,devo_senderus_01) -> Starting[EMERGENCYthread
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used insteadPERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T0826T09:5715:1801.964209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internalDevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCYStartingPERSISTENCEthreadSYSTEM](every 300 seconds)
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T08:57:18.964DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcessOutputProcess::MainThread -> Running custom validation rulesDevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T0826T09:5715:1801.964209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T0826T09:5715:1801.964209 INFO InputProcessOutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Creating API client. Starting thread (every 300 seconds)
2023-12-26T0826T09:5715:1801.964209 INFO InputProcessOutputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f2d62685640>DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T0826T09:5715:1801.964209 INFO OutputProcess::DevoSenderManager(lookupinternal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookupinternal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T0826T09:5715:1801.964209 INFO OutputProcess::MainThreadOutputInternalConsumer(internal_senders_consumer_0) -> DevoSenderManager [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders,manager,devo_usconsumer_10) -> Starting threadNothing retrieved from the persistence.
2023-12-26T0826T09:5715:1801.964210 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) Finalizing the execution of init_variables()Validating service input config
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T0826T09:5715:1801.964210 INFO OutputProcessInputProcess::DevoSenderManager(internal_senders,manager,devo_us_1)MainThread -> [EMERGENCYValidatingPERSISTENCEtheSYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.rate limiter config given by the user
2023-12-26T0826T09:5715:1801.964210 INFO OutputProcessInputProcess::OutputInternalConsumer(internal_senders_consumer_0)MainThread -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence<requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T0826T09:5715:1801.965211 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s) Adding raw config to the collector store
2023-12-26T0826T09:5715:1801.965211 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread (execution_period=60s)Running custom validation rules
2023-12-26T0826T09:5715:1801.965211 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Starting thread
Creating API client.
2023-12-26T0826T09:5715:1801.965211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) - Starting threadCreated request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T0826T09:5715:18.965 WARNING01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermittedmessagesDelivered,predefined) ->FinalizingWaitingtheuntilexecutionsetup will be executed
of init_variables()
2023-12-26T0826T09:5715:18.965 WARNING01.212 INFO InputProcess::ProofPointPullerSetupMainThread -> InputThread(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined,123456) ->The token/header/authentication has not been created yetStarting thread (execution_period=60s)
2023-12-26T0826T09:5715:1801.978212 INFO InputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.04MiB -> 40.04MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T08:57:18.979ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO OutputProcessInputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.52MiB -> 41.39MiB), VMS(926.00MiB -> 926.00MiB)ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T0826T09:5715:1901.718212 INFO OutputProcessInputProcess::DevoSender(internal_senders,devo_sender_0MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) ->CreatedStartinga sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139832919561120"
2023-12-26T08:57:22.226thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined):MainThread -> [GC] global: 28.0% -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2023-12-26T08:57:22.974 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Pull Started
2023-12-26T08:57:22.980 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 13:48:14+00:00 TO: 2023-12-21 14:48:14+00:00
2023-12-26T08:57:25.95428.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:01.713 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerProofPointPullerSetup(proofpoint_tap_collector,123456,clicksPermitted,predefinedproofpoint_tap#123456,messagesDelivered#predefined) -> Setup(Partial)forStatisticsmodulefor<ProofPointPuller>thishaspullbeencycle (@devo_pulling_id=1703561242970):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:25.955successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2025-03-05T15:59:40.797 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_topclicks#predefined) -> Pull Started
2025-03-05T15:59:40.799 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksPermitted,predefinedtap#1212090,people_topclicks#predefined) -> Time Window FROM: 20232025-1202-25 21 14:48:1455+00:00 TO: 20232025-1202-21251522:48:1455+00:00
20232025-1203-26T0805T15:5759:2744.948243 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksPermitted,predefinedtap#1212090,people_topclicks#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035612429701741170580790):Number of requests made: 21; Number of events received: 031; Number of duplicated events filtered out: 031; Number of events generated and sent: 0; Average of events per second: 0.000.
20232025-1203-26T0805T15:5759:2744.949243 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 15:48:14+00:00 TO: 2023-12-21 16:48:14+00:00
2023-12-26T08:57:29.784 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefinedtap#1212090,people_topclicks#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035612429701741170580790):Number of requests made: 31; Number of events received: 031; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:29.785 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 16:48:14+00:00 TO: 2023-12-21 17:48:14+00:00
2023-12-26T08:57:31.51531; Number of events generated and sent: 0; Average of events per second: 0.000.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
2025-03-05T15:59:44.243 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksPermitted,predefinedtap#1212090,people_topclicks#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035612429701741170580790):Number of requests made: 41; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; NumberAverage of duplicated events filteredperout: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:31.516 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 17:48:14+00:00 TO: 2023-12-21 18:48:14+00:00
2023-12-26T08:57:33.312 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 5; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:33.313 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 18:48:14+00:00 TO: 2023-12-21 19:48:14+00:00
2023-12-26T08:57:42.836 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
2023-12-26T08:57:42.836second: 0.000.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Expand
title
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
People VAP
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined)MainThread -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Expand
title
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
...
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
...
Error type
...
Error ID
...
Error message
...
Cause
...
Solution
...
SetupError
...
100
...
HTTP Error occurred while retrieving events from Proof point server
...
username and password is not correct
...
Make sure that credentials are correct.
...
101
...
Error occurred while retrieving events from ProofPoint server
...
start_time_in_utc is in future or not in proper format
...
Make sure the start time is not in future and not in proper format
...
PullError
...
300
...
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
...
This error happens when the collector tries to fetch the data from API.
...
In this error you will find the HTTP error code as well as the summary and details.
...
...
301
...
Some Error occurred while retrieving events from ProofPoint server : {Exception}
...
Some exception occured while making the API request.
...
Reach out to the developer with the exact error message.
Clicks blocked
Expand
title
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component
Description
Setup
The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
Code Block
ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) - Starting thread
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread (execution_period=60s)
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-21T1926T09:1015:1601.127209 INFO InputProcessOutputProcess::MainThread -> ProofPointPullerDevoSenderManager(proofpointinternal_tapsenders,123456,clicksPermitted,predefinedmanager,devo_us_1) -> Starting thread
2023-12-21T1926T09:1015:16.127 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined01.209 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> Waiting[EMERGENCYuntilPERSISTENCEsetup will be executed
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefinedSYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> The token/header/authentication has not been created yetNothing retrieved from the persistence.
2023-12-21T1926T09:1015:1601.127209 INFO InputProcessOutputProcess::MainThreadOutputInternalConsumer(internal_senders_consumer_0) -> ServiceThread(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread (execution_period=60s)[EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-21T1926T09:1015:1601.128210 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Starting threadValidating service input config
2023-12-21T1926T09:1015:1601.129210 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) - Starting threadRunning overriding rules
2023-12-21T1926T09:1015:16.129 WARNING01.210 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined)MainThread -> Validating theThe token/header/authentication has not been created yetrate limiter config given by the user
2023-12-21T1926T09:1015:16.130 WARNING01.210 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined)MainThread -> Waiting until setup<requests_limits> setting has not been defined. The generic settings will be executedused instead.
2023-12-21T1926T09:1015:1601.131211 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s) Adding raw config to the collector store
2023-12-21T1926T09:1015:1601.131211 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting threadRunning custom validation rules
2023-12-21T1926T09:1015:1601.131211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
Creating API client.
2023-12-21T1926T09:1015:16.131 WARNING01.211 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined)MainThread -> The token/header/authentication has not been created yetCreated request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-21T1926T09:1015:16.131 WARNING01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) ->FinalizingWaitingtheuntilexecutionsetup will be executedof init_variables()
2023-12-21T1926T09:1015:1601.145212 INFO InputProcess::MainThread -> [GC] global: 32.2% -> 32.2%, process: RSS(40.54MiB -> 40.54MiB), VMS(1.05GiB -> 1.05GiB)
InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-21T1926T09:1015:1601.768212 INFO OutputProcessInputProcess::DevoSender(internal_senders,devo_sender_0MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) ->Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140555598948768"
2023-12-21T19:10:18.447 INFOStarting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefinedmessagesDelivered#predefined) -> Setup for module <ProofPointPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2023-12-22T10:24:02.739The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::MainThread -> [GC] global::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> Pull Started
421.98MiB)
2023-12-22T1026T09:2415:0201.741713 INFO InputProcessOutputProcess::ProofPointPullerDevoSender(proofpointinternal_tap,123456,clicksBlocked,predefinedsenders,devo_sender_0) -> Time Window FROM: 2023-12-10 11:00:00+00:00 TO: 2023-12-10 12:00:00+00:00
2023-12-22T10:24:02.742 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Start_Time_in_utc must be at most 7.00d into the past, Changing the time startTime to be in the specified time
2023-12-22T10:24:04.567 Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerProofPointPullerSetup(proofpoint_tap_collector,123456,clicksBlocked,predefinedproofpoint_tap#123456,messagesDelivered#predefined) -> (Partial)SetupStatistics for thismodulepull<ProofPointPuller>cycle (@devo_pulling_id=1703220842731):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-22T10:24:04.568has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Info
Note that the PrePull action is executed only one time before the first run of the Pull action.
Code Block
2025-03-05T15:59:29.807 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_vap#predefined) -> Pull Started
2025-03-05T15:59:29.808 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksBlocked,predefinedtap#1212090,people_vap#predefined) -> Time Window FROM: 20232025-1202-15250521:5448:02.73167456+00:00 TO: 20232025-1202-15250622:5448:02.73167456+00:00
2023-12-22T10:24:06.17300023-12-26T09:15:12.140
2025-03-05T15:59:32.041 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksBlocked,predefinedtap#1212090,people_vap#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17032208427311741170569801):Number of requests made: 21; Number of events received: 0193; Number of duplicated events filtered out: 0187; Number of events generated and sent: 06; Average of events per second: 02.000687.
20232025-1203-22T1005T15:2459:0632.174041 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Time Window FROM: 2023-12-15 06:54:02.731674+00:00 TO: 2023-12-15 07:54:02.731674+00:00
2023-12-22T10:24:07.839 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefinedtap#1212090,people_vap#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17032208427311741170569801):Number of requests made: 31; Number of events received: 0193; Number of duplicated events filtered out: 0187; Number of events generated and sent: 06; Average of events per second: 02.000687.
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
Code Block
20232025-1203-26T0805T15:5759:4232.836041 INFO InputProcess::ProofPointPullerProofPointPeoplePuller(proofpoint_tap,123456,clicksBlocked,predefinedtap#1212090,people_vap#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=17035612429701741170569801):Number of requests made: 61; Number of events received: 0193; Number of duplicated events filtered out: 0187; Number of events generated and sent: 06; Average of events per second: 02.000687.
Info
The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
Expand
title
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start_time_in_utc_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Expand
title
Restart the persistenceTroubleshooting
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc
_format parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Expand
title
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
HTTP Error occurred while retrieving events from Proof point server
username and password is not correct
Make sure that credentials are correct.
101
Error occurred while retrieving events from ProofPoint server
start_time_in_utc is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Expand
title
Verify collector operations
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
Code Block
is in future or not in proper format
Make sure the start time is not in future and not in proper format
PullError
300
HTTP Error occurred while retrieving events from Proof point server : {summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
301
Some Error occurred while retrieving events from ProofPoint server : {Exception}
Some exception occured while making the API request.
Reach out to the developer with the exact error message.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Expand
title
Verify collector operations
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
Code Block
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:05:33.936 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.939 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:05:33.935941 INFO OutputProcessInputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting threadValidating the rate limiter config given by the user
2023-12-26T09:05:33.935941 INFO InputProcess::MainThread -> Validating<requests_limits> setting has not been defined module definition. The generic settings will be used instead.
2023-12-26T09:05:33.936941 INFO OutputProcessInputProcess::DevoSenderManager(internal_senders,manager,devo_us_1)MainThread -> [EMERGENCYAddingPERSISTENCErawSYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
config to the collector store
2023-12-26T09:05:33.936941 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> NothingCreatingretrieved from the persistenceAPI client.
2023-12-26T09:05:33.936941 INFO OutputProcessInputProcess::OutputLookupConsumer(lookup_senders_consumer_0)MainThread -> [EMERGENCYCreatedPERSISTENCErequestSYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f75040e44c0>
2023-12-26T09:05:33.939941 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Validating service input config
ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) Finalizing the execution of init_variables()
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
Code Block
2023-12-26T09:0520:3301.940210 INFO InputProcessOutputProcess::MainThreadDevoSenderManagerMonitor(internal_senders,devo_us_1) -> RunningNumberoverridingofrules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
available senders: 1, sender manager internal queue size: 0
2023-12-26T09:0520:3301.941210 INFO InputProcessOutputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.DevoSenderManagerMonitor(internal_senders,devo_us_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-12-26T09:0520:3301.941210 INFO InputProcessOutputProcess::MainThreadDevoSenderManagerMonitor(internal_senders,devo_us_1) -> Adding raw config to the collector storeSender: DevoSender(internal_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
2023-12-26T09:0520:3301.941209 INFO InputProcessOutputProcess::MainThreadDevoSenderManagerMonitor(standard_senders,devo_us_1) -> Running custom validation rules
Standard - Total number of messages: 0 messages/bytes sent since "2023-12-26T0926T03:0545:33.941 INFO InputProcess::MainThread -> Creating API client.01.203502+00:00": 0/0, (elapsed 0.000 seconds)
2023-12-26T09:0520:3301.941209 INFO InputProcessOutputProcess::MainThreadDevoSenderManagerMonitor(lookup_senders,devo_us_1) -> NumberCreatedofrequestavailableclientsenders: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f75040e44c0> 1, sender manager internal queue size: 0
2023-12-26T09:0520:3301.941209 INFO InputProcessOutputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) Finalizing the execution of init_variables()
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
Code Block
DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(internallookup_senders,devo_us_1) -> Number of available senders: 1, sender manager internal queue size: 0
Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_us_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Sender: DevoSender(internal_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
2023-12-26T09:20:01.209 INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Standard - Total number of messages: 0 messages/bytes sent since "2023-12-26T03:45:01.203502+00:00": 0/0, (elapsed 0.000 seconds)
2023-12-26T09:20:01.209 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-12-26T09:20:01.209 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2023-12-26T09:20:01.210::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Lookup - Total number of messages sent: 0, messages sent since "2023-12-26 03:45:01.205552+00:00": 0 (elapsed 0.000 seconds)
Info
By default, these information traces will be displayed every 10 minutes.
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:
Sender services
Description
internal_senders
In charge of delivering internal metrics to Devo such as logging traces or metrics.
standard_senders
In charge of delivering pulled events to Devo.
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace
Description
Number of available senders: 1
Displays the number of concurrent senders available for the given Sender Service.
sender manager internal queue size: 0
Displays the items available in the internal sender queue.
Info
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.
Total number of messages sent: 44, messages sent since "2023-11-10 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)
Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.
Info
By default these traces will be shown every 10 minutes.
...
Release
...
Released on
...
Release type
...
Details
...
Expand
title
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
Code Block
2023-12-26T08:57:18.978 INFO InputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.04MiB -> 40.04MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T08:57:18.979 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1)MainThread -> Lookup - Total number of messages sent: 0, messages sent since "2023-12-26 03:45:01.205552+00:00": 0 (elapsed 0.000 seconds)
Info
By default, these information traces will be displayed every 10 minutes.
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:
Sender services
Description
internal_senders
In charge of delivering internal metrics to Devo such as logging traces or metrics.
standard_senders
In charge of delivering pulled events to Devo.
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace
Description
Number of available senders: 1
Displays the number of concurrent senders available for the given Sender Service.
sender manager internal queue size: 0
Displays the items available in the internal sender queue.
Info
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.
Total number of messages sent: 44, messages sent since "2023-11-10 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)
Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.
Info
By default these traces will be shown every 10 minutes.
Expand
title
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
