Versions Compared

Old Version 2

changes.mady.by.user Juan Tomás Alonso Nieto

Saved on

compared with

New Version Current

changes.mady.by.user Juan Tomás Alonso Nieto

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Data source

Collector service

Optional

Flattening details

All

all_emails

yes

not required

Threat isolation

isolation

yes

not required

Malware

malware

yes

not required

clicktime

clicktime

yes

not required

Anti-spam

anti_spam

yes

not required

Email Threat Analytics

email_threat

yes

not required

Email delivery data

email_delivery

yes

not required

Vendor setup

-

Minimum configuration required for basic pulling

...

expand
Expand
titleVerify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

Code Block
2024-07-22T11:49:21.823    INFO InputProcess::MainThread -> ServiceThread(symantec,123123,threat_isolation,predefined) - Starting thread (execution_period=60s)
2024-07-22T11:49:21.823    INFO InputProcess::MainThread -> SymantecBasePullerSetup(unknown,symantec#123123,threat_isolation#predefined) -> Starting thread
2024-07-22T11:49:21.823    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-07-22T11:49:21.823    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-symantec/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-07-22T11:49:21.823    INFO InputProcess::MainThread -> SymantecBasePuller(symantec,123123,threat_isolation,predefined) - Starting thread
2024-07-22T11:49:21.823 WARNING InputProcess::SymantecBasePuller(symantec,123123,threat_isolation,predefined) -> Waiting until setup will be executed
2024-07-22T11:49:21.824    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-symantec/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-07-22T11:49:21.824    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-07-22T11:49:21.824    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-symantec/state/10dd360c86621afd5a28a029a0dddcf6"
2024-07-22T11:49:21.825    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-07-22T11:49:21.825    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2024-07-22T11:49:21.825    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-07-22T11:49:21.825    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-07-22T11:49:21.825    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-07-22T11:49:21.825    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-07-22T11:49:21.825    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
2024-07-22T11:49:21.826    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
2024-07-22T11:49:21.826    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-07-22T11:49:21.826    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-07-22T11:49:21.826    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-07-22T11:49:21.826    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-07-22T11:49:21.826    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-07-22T11:49:21.826    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-07-22T11:49:21.827    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
2024-07-22T11:49:21.827    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
2024-07-22T11:49:21.827    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-07-22T11:49:21.827    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-07-22T11:49:21.827    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-07-22T11:49:21.827    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-07-22T11:49:21.829    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-07-22T11:49:21.837    INFO InputProcess::MainThread -> [GC] global: 26.6% -> 26.7%, process: RSS(42.00MiB -> 42.00MiB), VMS(496.55MiB -> 496.55MiB)
2024-07-22T11:49:21.842    INFO OutputProcess::MainThread -> [GC] global: 26.7% -> 26.7%, process: RSS(43.10MiB -> 43.47MiB), VMS(929.06MiB -> 929.06MiB)
2024-07-22T11:49:22.290    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/md_tausif/gitlab/devo-collector-symantec/certs/chain.crt", "cert_path": "/home/md_tausif/gitlab/devo-collector-symantec/certs/int-if-integrations-india.crt", "key_path": "/home/md_tausif/gitlab/devo-collector-symantec/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "140337000446320"
2024-07-22T11:49:22.290    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-07-22T11:49:23.198    INFO InputProcess::SymantecBasePullerSetup(unknown,symantec#123123,threat_isolation#predefined) -> Setup for module <SymantecBasePuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Info

Note that the PrePull action is executed only one time before the first run of the Pull action.

Code Block
2024-07-22T11:49:23.835    INFO InputProcess::SymantecBasePuller(symantec,123123,threat_isolation,predefined) -> Pull Started
2024-07-22T11:49:23.836    INFO InputProcess::SymantecBasePuller(symantec,123123,threat_isolation,predefined) -> Fetching data From : 2024-07-22T04:00:00Z to the current date
2024-07-22T11:49:30.201    INFO InputProcess::SymantecBasePuller(symantec,123123,threat_isolation,predefined) -> Sent 1000 threat_isolation events to Devo.
2024-07-22T11:49:30.201    INFO InputProcess::SymantecBasePuller(symantec,123123,threat_isolation,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1721629163826):Number of requests made: 2; Number of events received: 1000; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000; Average of events per second: 157.095.

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Code Block
2024-07-22T11:49:38.664    INFO InputProcess::SymantecBasePuller(symantec,123123,threat_isolation,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1721629163826):Number of requests made: 4; Number of events received: 1516; Number of duplicated events filtered out: 0; Number of events generated and sent: 1516; Average of events per second: 102.231.
title
Restart the 
...
persistance
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
  1. Edit the configuration file.
  2. Change the value of the request_period_in_seconds parameter to a different one.
  3. Save the changes.
  4. Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
title
 Note
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
 Expand
Troubleshooting
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
Error type
Error ID
Error message
Cause
Solution
SetupError
100
Error occurred while requesting from the Symantec server. Error message: {error message}
Credentials error {unauthorized} or mismatch b/w credentials or Symantec server error
Check the credentials or contact developer with required message
SetupError
101
Some error occurred while retrieving events from symantec server. Error detail {e}.
Error on the symantec server.
Contact the developer with exact error message.
PullError
300
Error related to HTTP, occurred while retrieving events from symantec server{summery} , {details}
This error happens when the collector tries to fetch the data from API.
In this error you will find the HTTP error code as well as the summary and details.
PullError
301
Some error occurred while retrieving events from symantec server. Error details: {details}
Some exceptions occurred while making the API request.
Contact the developer with exact error message.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
...