| title | Internal process and deduplication methodProofpoint Isolation API stores events up to 30 days in the past. All isolation events are fetched on a polling basis. If there are more events available for a given time period than can be retrieved in a single page (limited to 10,000 records per page), the collector continues to fetch new events until none are available for the query window. Events are sorted by the date field and deduplicated by comparing the hash signature of the event against previous pulls. Devo categorization and destinationAll events of this service are ingested into the table tap.proofpoint.isolation.url_isolation. Setup outputA successful run has the following output messages for the setup module: | Code Block |
|---|
2023-04-03T16:18:50.783 INFO InputProcess::MainThread -> IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> Starting thread
2023-04-03T16:18:50.784 WARNING InputProcess::IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> The token/header/authentication has not been created yet
2023-04-03T16:18:50.785 INFO InputProcess::IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> Making sample request to test authentication.
2023-04-03T16:18:50.792 INFO InputProcess::MainThread -> IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> Starting thread
2023-04-03T16:18:50.792 INFO InputProcess::MainThread -> IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) - Starting thread
2023-04-03T16:18:50.792 WARNING InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Waiting until setup will be executed
2023-04-03T16:18:50.793 WARNING InputProcess::IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> The token/header/authentication has not been created yet
2023-04-03T16:18:51.509 INFO InputProcess::IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> Setup for module <IsolationPuller> has been successfully executed
2023-04-03T16:18:51.510 INFO InputProcess::IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> Making sample request to test authentication.
2023-04-03T16
2023-04-03T16:20:12.074 INFO InputProcess::IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> Authentication successful.
2023-04-03T16:18:52.096 INFO InputProcess::IsolationPullerSetup(unknown,proofpoint_isolation#10001,url_isolation#predefined) -> Setup for module <IsolationPuller> has been successfully executed |
Puller output| Code Block |
|---|
2023-05-16T23:02:54.695 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) Starting the execution of pre_pull()
2023-05-16T23:02:54.696 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Reading persisted data
2023-05-16T23:02:54.698 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Data retrieved from the persistence: {'start_time_in_utc': '2023-05-16T00:00:00.000000Z', 'last_event_time_in_utc': '2023-05-17T03:01:40.121516Z', 'last_ids': ['57bddd353b7cc010ab4a696cf86d4c18e97aafbdc49a52498bddc756de044c9f', '57bddd353b7cc010ab4a696cf86d4c18e97aafbdc49a52498bddc756de044c9f'], '@persistence_version': 1}
2023-05-16T23:02:54.699 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Running the persistence upgrade steps
2023-05-16T23:02:54.700 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Running the persistence corrections steps
2023-05-16T23:02:54.700 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Running the persistence corrections steps
2023-05-16T23:02:54.701 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> No changes were detected in the persistence
2023-05-16T23:02:54.702 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) Finalizing the execution of pre_pull()
2023-05-16T23:02:54.702 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Starting data collection every 60 seconds
2023-05-16T23:02:54.702 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Pull Started
2023-05-16T23:02:54.702 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Fetching all url_isolation event(s) from 2023-05-17T03:01:40.121516Z to 2023-05-17T03:02:54.695673Z.
2023-05-16T23:02:55.648 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> Updating the persistence
2023-05-16T23:02:55.650 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1684292574695):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. |
After a successful collector’s execution (that is, no error logs were found), you should be able to see the following log message: | Code Block |
|---|
2023-04-03T16:20:41.575 INFO InputProcess::IsolationPuller(proofpoint_isolation,10001,url_isolation,predefined) -> The data is up to date! |
|
|---|
