Proofpoint Tap collector
Juan Tomás Alonso Nieto (Deactivated)
MD Tausif
Virginia Camuñas Núñez
Borja Moro Moreno (Unlicensed)
Overview
Proofpoint Targeted Attack Protection (TAP) helps you stay ahead of attackers with an innovative approach that detects, analyzes and blocks advanced threats.
Devo collector features
Feature | Details |
|---|
Feature | Details |
|---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Data sources
Data source | Description | API endpoint | Collector service name | Devo table | Available from release |
|---|
Data source | Description | API endpoint | Collector service name | Devo table | Available from release |
|---|---|---|---|---|---|
| Fetch events for clicks to malicious URLs blocked in the specified time period |
|
|
|
|
| Fetch events for clicks to malicious URLs permitted in the specified time period |
|
|
|
|
| Fetch events for messages blocked in the specified time period that contained a known threat. |
|
|
|
|
| Fetch events for messages delivered in the specified time period which contained a known threat. |
|
|
|
|
| The Threats API allows administrators to pull detailed attributes about individual threats observed in their environment. It can be used to retrieve more intelligence for threats identified in the SIEM or Campaign API responses |
|
|
|
|
| If Forensics is enabled the events are flattened into the table |
|
|
|
|
| The Campaign API allows administrators to pull campaign IDs in a timeframe and specific details about campaigns, including: their description; the actor, malware family, and techniques associated with the campaign; and the threat variants which have been associated with the campaign |
|
|
|
|
| If Forensics is enabled the events are flattened into the table |
|
|
|
|
| The People API allows administrators to identify which users in their organizations were most attacked or are the top clickers during a specified period. Fetch the identities and attack index of the top clickers within your organization for a given period. Top clickers are the users who have demonstrated a tendency to click on malicious URLs, regardless of whether the clicks were blocked or not. Knowing who are more susceptible to threats is useful for proactive security approaches such as security training assignments. |
|
|
|
|
| The People API allows administrators to identify which users in their organizations were most attacked or are the top clickers during a specified period. Fetch the identities and attack index breakdown of Very Attacked People within your organization for a given period. |
|
|
|
|
For more information on how the events are parsed, visit our page.
Flattening preprocessing
Data source | Collector service | Optional | Flattening details |
|---|
Data source | Collector service | Optional | Flattening details |
|---|---|---|---|
|
|
| not required |
|
|
| not required |
Accepted authentication methods
Authentication method | username | password |
|---|
Authentication method | username | password |
|---|---|---|
| REQUIRED | REQUIRED |
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting | Details |
|---|
Setting | Details |
|---|---|
| The username for proofpoint Tap |
| The password(credential) for proofpoint |
| Start Time which is not more than 7 days into the past |
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.
API Limits, issues
for services clicksBlocked, clicksPermitted, messageBlocked, messageDelivered
ClicksBlocked(/v2/siem/clicks/blocked) ,messageBlocked(/v2/siem/messages/blocked),messageDelivered(/v2/siem/messages/delivered) can collectively make 1800 requests per day as per api documentationclicksPermitted(/v2/siem/clicks/permitted) can make 1800 reuqests per day as per api documentation.To simplify the code and handle all the four services, code has been written in a way that each of the four service can make 220 request per day.
We have put a limit on the api call, which will block the api requests after 220 requests for a service.
If the collector took, let’s say, 15 hours to make 220 requests, the collector will block the api call for 9 hours (9 + 15 = 24 hours) and it will resume making api requests after 9 hours. For these 9 hours no ingestion will occur for that service.
Issues:
Let’s say all the services make 1000 request in 15 hours and then due to some issue collector restarts. Now, after the restart collector is going to assume that service still make a 1800 request , ignoring the 1000 requests collector made before the restart. This will lead to surpass the 1800 requets per day api limit causing a 429 error.
for services threats, campaigns, people_topclicks, people_vap
threatsservice has no api limitpeople_topclicks(v2/people/top-clickers) andpeople_vap(v2/people/vap) services can make only 50 requests in 1 day as per the api documentationcampaigns(v2/campaign/ids) service can only make 50 request in 1 day as per the api documentationWe have put a limit on the api call, which will block the api requests after 50 requests for a service
If the collector took , let’s say, 10 hours to make 50 request, the collector will block the api call for 14 hours (10 + 14 = 24 hours) and it will resume making api requests after 14 hours. For these 14 hours no ingestion will occur for that service
Issues:
Let’s say a service make 30 request in 3 hours and then due to some issue collector restarts. Now, after the restart collector is going to assume that service still make a 50 request , ignoring the 30 requests collector made before the restart. This will lead to surpass the 50 requets per day api limit causing a 429 error for that service.
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Messages blocked
No. of request this service can make in a day is 220.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-12-26T09:05:33.934 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:05:33.935 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:05:33.936 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.939 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f75040e44c0>
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) Finalizing the execution of init_variables()
2023-12-26T09:05:33.942 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:05:33.942 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:05:33.942 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Starting thread
2023-12-26T09:05:33.942 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:05:33.943 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread
2023-12-26T09:05:33.943 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Waiting until setup will be executed
2023-12-26T09:05:33.947 INFO OutputProcess::MainThread -> [GC] global: 28.2% -> 28.2%, process: RSS(40.77MiB -> 41.64MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:05:33.954 INFO InputProcess::MainThread -> [GC] global: 28.2% -> 28.2%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:05:34.739 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140140610316704"
2023-12-26T09:05:36.819 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
023-12-26T09:05:36.953 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Pull Started
2023-12-26T09:05:36.956 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-10 11:00:00+00:00 TO: 2023-12-10 12:00:00+00:00
2023-12-26T09:05:36.956 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Start_Time_in_utc must be at most 7.00d into the past, Changing the time startTime to be in the specified time
2023-12-26T09:05:39.320 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:39.321 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 04:35:36.946633+00:00 TO: 2023-12-19 05:35:36.946633+00:00
2023-12-26T09:05:41.780 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:41.781 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 05:35:36.946633+00:00 TO: 2023-12-19 06:35:36.946633+00:00
2023-12-26T09:05:44.135 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:05:44.136 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Time Window FROM: 2023-12-19 06:35:36.946633+00:00 TO: 2023-12-19 07:35:36.946633+00:00
2023-12-26T09:05:46.495 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 4; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 0.105.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2023-12-26T09:05:46.495 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561736946):Number of requests made: 4; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 1; Average of events per second: 0.105.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
SetupError | 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
| start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | |
PullError | 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
Messages delivered
No. of request this service can make in a day is 220.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:01.713 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2023-12-26T09:15:04.224 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Pull Started
2023-12-26T09:15:04.227 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 03:00:00+00:00 TO: 2023-12-25 04:00:00+00:00
2023-12-26T09:15:12.140 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:15:12.140 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 04:00:00+00:00 TO: 2023-12-25 05:00:00+00:00
2023-12-26T09:15:13.985 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:15:13.986 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 05:00:00+00:00 TO: 2023-12-25 06:00:00+00:00
2023-12-26T09:15:16.077 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T09:15:16.077 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Time Window FROM: 2023-12-25 06:00:00+00:00 TO: 2023-12-25 07:00:00+00:00
2023-12-26T09:15:18.287 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703562304216):Number of requests made: 4; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2023-12-22T10:28:58.153 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 164; Number of events received: 14; Number of duplicated events filtered out: 0; Number of events generated and sent: 14; Average of events per second: 0.047.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
SetupError | 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
| start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | |
PullError | 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
Clicks permitted
No. of request this service can make in a day is 220.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-12-26T08:57:18.936 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T08:57:18.957 WARNING InputProcess::MainThread -> A previous rate limiter with same "period_in_seconds" and "number_of requests" was already existing: "86400/1800"
2023-12-26T08:57:18.957 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) Starting the execution of init_variables()
2023-12-26T08:57:18.957 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T08:57:18.959 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T08:57:18.962 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T08:57:18.963 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T08:57:18.963 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.963 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T08:57:18.963 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.963 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T08:57:18.964 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T08:57:18.964 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T08:57:18.964 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f2d62685640>
2023-12-26T08:57:18.964 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.964 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T08:57:18.964 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) Finalizing the execution of init_variables()
2023-12-26T08:57:18.964 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.964 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread (execution_period=60s)
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Starting thread
2023-12-26T08:57:18.965 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread
2023-12-26T08:57:18.965 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Waiting until setup will be executed
2023-12-26T08:57:18.965 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> The token/header/authentication has not been created yet
2023-12-26T08:57:18.978 INFO InputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.04MiB -> 40.04MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T08:57:18.979 INFO OutputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.52MiB -> 41.39MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T08:57:19.718 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139832919561120"
2023-12-26T08:57:22.226 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2023-12-26T08:57:22.974 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Pull Started
2023-12-26T08:57:22.980 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 13:48:14+00:00 TO: 2023-12-21 14:48:14+00:00
2023-12-26T08:57:25.954 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:25.955 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 14:48:14+00:00 TO: 2023-12-21 15:48:14+00:00
2023-12-26T08:57:27.948 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:27.949 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 15:48:14+00:00 TO: 2023-12-21 16:48:14+00:00
2023-12-26T08:57:29.784 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:29.785 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 16:48:14+00:00 TO: 2023-12-21 17:48:14+00:00
2023-12-26T08:57:31.515 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 4; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:31.516 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 17:48:14+00:00 TO: 2023-12-21 18:48:14+00:00
2023-12-26T08:57:33.312 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 5; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-26T08:57:33.313 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Time Window FROM: 2023-12-21 18:48:14+00:00 TO: 2023-12-21 19:48:14+00:00
2023-12-26T08:57:42.836 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2023-12-26T08:57:42.836 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
SetupError | 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
| start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | |
PullError | 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
Clicks blocked
No. of request this service can make in a day is 220.
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) - Starting thread
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread (execution_period=60s)
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Starting thread
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) - Starting thread
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksPermitted,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.127 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> The token/header/authentication has not been created yet
2023-12-21T19:10:16.127 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread (execution_period=60s)
2023-12-21T19:10:16.128 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> Starting thread
2023-12-21T19:10:16.129 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) - Starting thread
2023-12-21T19:10:16.129 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesBlocked#predefined) -> The token/header/authentication has not been created yet
2023-12-21T19:10:16.130 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.131 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-21T19:10:16.131 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-21T19:10:16.131 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-21T19:10:16.131 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-21T19:10:16.131 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-21T19:10:16.145 INFO InputProcess::MainThread -> [GC] global: 32.2% -> 32.2%, process: RSS(40.54MiB -> 40.54MiB), VMS(1.05GiB -> 1.05GiB)
2023-12-21T19:10:16.768 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140555598948768"
2023-12-21T19:10:18.447 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,clicksPermitted#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2023-12-22T10:24:02.739 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Pull Started
2023-12-22T10:24:02.741 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Time Window FROM: 2023-12-10 11:00:00+00:00 TO: 2023-12-10 12:00:00+00:00
2023-12-22T10:24:02.742 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Start_Time_in_utc must be at most 7.00d into the past, Changing the time startTime to be in the specified time
2023-12-22T10:24:04.567 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-22T10:24:04.568 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Time Window FROM: 2023-12-15 05:54:02.731674+00:00 TO: 2023-12-15 06:54:02.731674+00:00
2023-12-22T10:24:06.173 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 2; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2023-12-22T10:24:06.174 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> Time Window FROM: 2023-12-15 06:54:02.731674+00:00 TO: 2023-12-15 07:54:02.731674+00:00
2023-12-22T10:24:07.839 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703220842731):Number of requests made: 3; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2023-12-26T08:57:42.836 INFO InputProcess::ProofPointPuller(proofpoint_tap,123456,clicksBlocked,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1703561242970):Number of requests made: 6; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
| 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
|
| Make sure the start time is not in future and not in proper format | |
| 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
Threats
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,threats,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,threats,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,threats#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,threats#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:01.713 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,threats#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2025-03-05T16:28:00.760 INFO InputProcess::ProofPointThreatsPuller(proofpoint_tap#1212090,threats#predefined) -> Pull Started
2025-03-05T16:28:00.762 INFO InputProcess::ProofPointThreatsPuller(proofpoint_tap#1212090,threats#predefined) -> Time Window FROM: 2024-09-26 14:00:00+00:00 TO: 2024-09-26 15:00:00+00:00
2025-03-05T16:59:32.041 INFO InputProcess::ProofPointThreatsPuller(proofpoint_tap#1212090,threats#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1741170569801):Number of requests made: 1; Number of events received: 193; Number of duplicated events filtered out: 187; Number of events generated and sent: 6; Average of events per second: 2.687.
2025-03-05T16:59:32.041 INFO InputProcess::ProofPointThreatsPuller(proofpoint_tap#1212090,threats#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741170569801):Number of requests made: 1; Number of events received: 193; Number of duplicated events filtered out: 187; Number of events generated and sent: 6; Average of events per second: 2.687.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2025-03-05T16:59:32.041 INFO InputProcess::ProofPointThreatsPuller(proofpoint_tap#1212090,threats#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741170569801):Number of requests made: 1; Number of events received: 193; Number of duplicated events filtered out: 187; Number of events generated and sent: 6; Average of events per second: 2.687.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
SetupError | 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
| start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | |
PullError | 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
Campaigns
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:01.713 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2025-03-05T16:28:01.769 INFO InputProcess::ProofPointCampaignsPuller(proofpoint_tap#1212090,campaigns#predefined) -> Pull Started
2025-03-05T16:28:01.770 INFO InputProcess::ProofPointCampaignsPuller(proofpoint_tap#1212090,campaigns#predefined) -> Time Window FROM: 2025-02-25 11:30:29+00:00 TO: 2025-02-26 11:29:29+00:00
2025-03-05T16:28:07.447 INFO InputProcess::ProofPointCampaignsPuller(proofpoint_tap#1212090,campaigns#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1741172281758):Number of requests made: 1; Number of events received: 359; Number of duplicated events filtered out: 359; Number of events generated and sent: 0; Average of events per second: 0.000.
2025-03-05T16:28:07.447 INFO InputProcess::ProofPointCampaignsPuller(proofpoint_tap#1212090,campaigns#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741172281758):Number of requests made: 1; Number of events received: 359; Number of duplicated events filtered out: 359; Number of events generated and sent: 0; Average of events per second: 0.000.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2025-03-05T16:28:07.447 INFO InputProcess::ProofPointCampaignsPuller(proofpoint_tap#1212090,campaigns#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741172281758):Number of requests made: 1; Number of events received: 359; Number of duplicated events filtered out: 359; Number of events generated and sent: 0; Average of events per second: 0.000.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
SetupError | 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
| start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | |
PullError | 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
People Topclicks
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:01.713 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2025-03-05T15:59:40.797 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_topclicks#predefined) -> Pull Started
2025-03-05T15:59:40.799 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_topclicks#predefined) -> Time Window FROM: 2025-02-25 21:48:55+00:00 TO: 2025-02-25 22:48:55+00:00
2025-03-05T15:59:44.243 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_topclicks#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1741170580790):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.
2025-03-05T15:59:44.243 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_topclicks#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741170580790):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2025-03-05T15:59:44.243 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_topclicks#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741170580790):Number of requests made: 1; Number of events received: 31; Number of duplicated events filtered out: 31; Number of events generated and sent: 0; Average of events per second: 0.000.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
SetupError | 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
| start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | |
PullError | 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
People VAP
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
|---|
Component | Description |
|---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2023-12-26T09:15:01.181 INFO OutputProcess::MainThread -> Process started
2023-12-26T09:15:01.182 INFO MainProcess::MainThread -> Started all object from "MainProcess" process
2023-12-26T09:15:01.182 INFO InputProcess::MainThread -> Process Started
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Starting the execution of init_variables()
2023-12-26T09:15:01.204 INFO InputProcess::MainThread -> Validating service metadata
2023-12-26T09:15:01.206 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:15:01.207 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.208 INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputStandardConsumer(standard_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(standard_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(lookup_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Starting thread (every 300 seconds)
2023-12-26T09:15:01.209 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:15:01.209 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.209 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:15:01.210 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f364f0fe5e0>
2023-12-26T09:15:01.211 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) Finalizing the execution of init_variables()
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> InputThread(proofpoint_tap,123456) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ServiceThread(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread (execution_period=60s)
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Starting thread
2023-12-26T09:15:01.212 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) - Starting thread
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPuller(proofpoint_tap,123456,messagesDelivered,predefined) -> Waiting until setup will be executed
2023-12-26T09:15:01.212 WARNING InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> The token/header/authentication has not been created yet
2023-12-26T09:15:01.221 INFO OutputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.77MiB -> 41.89MiB), VMS(926.00MiB -> 926.00MiB)
2023-12-26T09:15:01.225 INFO InputProcess::MainThread -> [GC] global: 28.0% -> 28.1%, process: RSS(40.29MiB -> 40.29MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T09:15:01.713 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/chain.crt", "cert_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.crt", "key_path": "/home/mdtausif/Gitlab/devo-collector-proofpoint-tap/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "139871239825152"
2023-12-26T09:15:03.324 INFO InputProcess::ProofPointPullerSetup(proofpoint_tap_collector,proofpoint_tap#123456,messagesDelivered#predefined) -> Setup for module <ProofPointPuller> has been successfully executedPuller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2025-03-05T15:59:29.807 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_vap#predefined) -> Pull Started
2025-03-05T15:59:29.808 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_vap#predefined) -> Time Window FROM: 2025-02-25 21:48:56+00:00 TO: 2025-02-25 22:48:56+00:00023-12-26T09:15:12.140
2025-03-05T15:59:32.041 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_vap#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1741170569801):Number of requests made: 1; Number of events received: 193; Number of duplicated events filtered out: 187; Number of events generated and sent: 6; Average of events per second: 2.687.
2025-03-05T15:59:32.041 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_vap#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741170569801):Number of requests made: 1; Number of events received: 193; Number of duplicated events filtered out: 187; Number of events generated and sent: 6; Average of events per second: 2.687.After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2025-03-05T15:59:32.041 INFO InputProcess::ProofPointPeoplePuller(proofpoint_tap#1212090,people_vap#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1741170569801):Number of requests made: 1; Number of events received: 193; Number of duplicated events filtered out: 187; Number of events generated and sent: 6; Average of events per second: 2.687.The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc_formatparameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type | Error ID | Error message | Cause | Solution |
|---|
Error type | Error ID | Error message | Cause | Solution |
|---|---|---|---|---|
SetupError | 100 |
| username and password is not correct | Make sure that credentials are correct. |
101 |
| start_time_in_utc is in future or not in proper format | Make sure the start time is not in future and not in proper format | |
PullError | 300 |
| This error happens when the collector tries to fetch the data from API. | In this error you will find the HTTP error code as well as the summary and details. |
| 301 |
| Some exception occured while making the API request. | Reach out to the developer with the exact error message. |
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
2023-12-26T09:05:33.935 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_us_1) -> Starting thread
2023-12-26T09:05:33.935 INFO InputProcess::MainThread -> Validating defined module definition
2023-12-26T09:05:33.936 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_us_1) -> [EMERGENCY PERSISTENCE SYSTEM] DevoSenderManager(internal_senders,manager,devo_us_1) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputInternalConsumer(internal_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.936 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY PERSISTENCE SYSTEM] OutputLookupConsumer(lookup_senders_consumer_0) -> Nothing retrieved from the persistence.
2023-12-26T09:05:33.939 INFO InputProcess::MainThread -> Validating common input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Validating service input config
2023-12-26T09:05:33.940 INFO InputProcess::MainThread -> Running overriding rules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Validating the rate limiter config given by the user
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> <requests_limits> setting has not been defined. The generic settings will be used instead.
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Adding raw config to the collector store
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Running custom validation rules
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Creating API client.
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> Created request client: <agent.modules.proofpoint.commons.proofpoint_client.ProofPointClient object at 0x7f75040e44c0>
2023-12-26T09:05:33.941 INFO InputProcess::MainThread -> ProofPointPuller(proofpoint_tap,123456,messagesBlocked,predefined) Finalizing the execution of init_variables()Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_us_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(internal_senders,devo_us_1) -> Sender: DevoSender(internal_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
2023-12-26T09:20:01.209 INFO OutputProcess::DevoSenderManagerMonitor(standard_senders,devo_us_1) -> Standard - Total number of messages: 0 messages/bytes sent since "2023-12-26T03:45:01.203502+00:00": 0/0, (elapsed 0.000 seconds)
2023-12-26T09:20:01.209 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Number of available senders: 1, sender manager internal queue size: 0
2023-12-26T09:20:01.209 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> enqueued_elapsed_times_in_seconds_stats: {}
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Sender: DevoSender(lookup_senders,devo_sender_0), status: {"internal_queue_size": 0, "is_connection_open": False}
2023-12-26T09:20:01.210 INFO OutputProcess::DevoSenderManagerMonitor(lookup_senders,devo_us_1) -> Lookup - Total number of messages sent: 0, messages sent since "2023-12-26 03:45:01.205552+00:00": 0 (elapsed 0.000 seconds)By default, these information traces will be displayed every 10 minutes.
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:
Sender services | Description |
|---|
Sender services | Description |
|---|---|
| In charge of delivering internal metrics to Devo such as logging traces or metrics. |
| In charge of delivering pulled events to Devo. |
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace | Description |
|---|
Logging trace | Description |
|---|---|
| Displays the number of concurrent senders available for the given Sender Service. |
| Displays the items available in the internal sender queue. This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
| Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
By default these traces will be shown every 10 minutes. |
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the
globalvalue.All metrics (Global, RSS, VMS) include the value before freeing and after
previous -> after freeing memory
2023-12-26T08:57:18.978 INFO InputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.04MiB -> 40.04MiB), VMS(421.98MiB -> 421.98MiB)
2023-12-26T08:57:18.979 INFO OutputProcess::MainThread -> [GC] global: 28.6% -> 28.7%, process: RSS(40.52MiB -> 41.39MiB), VMS(926.00MiB -> 926.00MiB)Differences between RSS and VMS memory usage:
RSSis the Resident Set Size, which is the actual physical memory the process is usingVMSis the Virtual Memory Size which is the virtual memory that process is using
Change log
Release | Released on | Release type | Recommendations |
|---|
Release | Released on | Release type | Recommendations |
|---|---|---|---|
| Oct 29, 2024 | bug fixing |
|
Bugs
| |||
| Oct 24, 2024 | IMPROVEMENTS |
|
Improvements
Bugs
| |||
| Aug 26, 2024 | IMPROVEMENTS |
|
Improvements
New features
| |||
| Feb 23, 2024 | IMPROVEMENTS |
|
Improvements
| |||
| Jan 16, 2024 | IMPROVEMENTS |
|
Improvements
Bugs
| |||
| Dec 21, 2023 | FIRST RELEASE |
|
Released the first version of the Proofpoint TAP collector. | |||