| Table of Contents | ||||||
|---|---|---|---|---|---|---|
|
...
Kiteworks APIs provide broad coverage of the platform. The APIs can be categorized into Content, Collaboration, Preferences, Contacts, Security, Clients, and Kiteworks Maintenance APIs.
Devo collector features
Feature | Details |
|---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Allowed source events obfuscation |
|
Data sources
Data source | API endpoint | Collector service name | Devo table |
|---|---|---|---|
Admin |
|
|
|
For more information on how the events are parsed, visit our page ← LINK TO THE PARSER ARTICLE IF EXISTS.
Vendor setup
-
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
| Info |
|---|
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details. |
Setting | Details |
|---|---|
| - |
| Info |
|---|
See the Accepted authentication methods section to verify what settings are required based on the desired authentication method. |
Accepted authentication methods
-
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
...
| Rw tab | ||
|---|---|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
| Code Block |
|---|
<any_directory>
└── devo-collectors/
└── <product_name>/
├── certs/
│ ├── chain.crt
│ ├── <your_domain>.key
│ └── <your_domain>.crt
├── state/
└── config/
└── config.yaml |
| Note |
|---|
Replace |
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.
...
| Note |
|---|
Replace |
Editing the config.yaml file
...
Getting Credentials
To log in to the kiteworks environment. Using the vendor doc here:
Enable the Kiteworks API Playground
The following steps help you get started with the Kiteworks API playground. Exploring using Kiteworks APIs requires development experience. To enable the Kiteworks API Playground UI:
In the new Kiteworks Admin console, go to Applications > Client Management > Custom Applications.
Turn on Enable Kiteworks API Playground UI. The Kiteworks Developer Documentation is added to the Help menu.
To view the complete list of APIs, go to the Help (?) menu and click Kiteworks Developer Documentation. The Developer Documentation page displays listing the library of APIs.
...
Setting Up Credentials
Create a custom application (ensuring that Signature Authorization) is enabled.
Go to the playground at https:///rest/index.html.
On the Kiteworks API Documentation toolbar, click the Get a Token button.
In the Request OAuth Token dialog box, select Signature-based Access Token from the grant list.
Fill in the information based on the application you just created in the administrator console.
Test all the API endpoints through the playground.
...
If you have not already done so, register at https://developer.kiteworks.com .
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
| Info |
|---|
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details. |
Setting | Details |
|---|---|
| The Kiteworks client ID |
| The Kiteworks client secret |
| The Kiteworks signature secret |
| The Kiteworks user email secret |
| Add your domain to the the url |
| Add your domain to the url |
Accepted authentication methods
Authentication Method | Client ID | Client Secret | Signature Secret | User Email |
|---|---|---|---|---|
Signature-based Access Token | Required | Required | Required | Required |
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
...
Error type
...
Error ID
...
Error message
...
Cause
...
Solution
...
SetupError
...
1
...
api_base_url must be specified
...
The api_base_url setting is missing.
...
Make sure the api_base_url setting is present under the events service in your configuration.
...
2
...
api_base_url not of expected type: str
...
The api_base_url setting has a type other than string.
...
Make sure the api_base_url setting is a string.
...
3
...
api_base_url must match one of two regexes: [...]
...
The api_base_url setting does not follow the expected format.
...
Make sure your api_base_url has this format: http[s]://{ip_address_or_domain}{:optional_port}
...
4
...
Required setting, credentials not found in user configuration
...
There is no credentials section in your input settings.
...
Make sure there is a credentials section under the threatquotient_data_puller input in your configuration.
...
5
...
Required setting, credentials not of expected type: dict
...
The credentials section is empty or has a simple type (is not an object).
...
Make sure the credentials section has a username and password fields.
...
6
...
Required setting, username not found in user configuration
...
The username setting is missing.
...
Make sure the username setting from the credentials section has a value.
...
7
...
Required setting, username not of expected type: str
...
The username setting has a type other than string.
...
Make sure the username setting from the credentials section is a string.
...
8
...
Required setting, password not found in user configuration
...
The password setting is missing.
...
Make sure the password setting from the credentials section has a value.
...
9
...
Required setting, password not of expected type: str
...
The password setting has a type other than string.
...
Make sure the password setting from the credentials section is a string.
...
10
...
Optional setting, verify_host_ssl_cert not of expected type: bool
...
The verify_host_ssl_cert setting has a type other than boolean.
...
Make sure the verify_host_ssl_cert setting is a boolean value (true/false).
...
11
...
event_fetch_limit_in_items must be greater than or equal to [...] and less than equal to [...]
...
The event_fetch_limit_in_items setting has a value too low or too high for the specified limits.
...
Make sure the event_fetch_limit_in_items setting is an integer ranged between the specified limits.
...
12
...
devo_tag_map must have an entry named "default"
...
This error is not expected to happen in a regular flow.
...
This needs to be troubleshooted by the colllector’s developers.
...
13
...
Required setting, reset_persistence_auth not of expected type: str
...
The reset_persistence_auth setting has a value, but its type is other than string.
...
Make sure the reset_persistence_auth setting is a string.
...
14
...
Required setting, historical_poll_datetime not of expected type: str
...
The historical_poll_datetime setting has a type other than string.
...
Make sure the historical_poll_datetime setting is a string.
...
15
...
historical_poll_datetime does not match expected format [...]
...
The historical_poll_datetime setting does not look like a valid date.
...
Make sure the historical_poll_datetime setting meets the mentioned format (a reference of this representation can be found here).
...
16
...
Please enter valid date for historical_poll_datetime less than or equal to one year
...
The historical_poll_datetime setting is a date older than one year.
...
Make sure the historical_poll_datetime setting does not represent a date older than one year.
...
17
...
Please enter valid date for historical_poll_datetime less than or equal to the current date
...
The historical_poll_datetime setting is a future date.
...
Make sure the historical_poll_datetime setting does not represent a future date.
...
InitVariablesError
...
100
...
Unexpected status code when fetching ThreatQuotient JWT: [...]
...
When a token was retrieved, the response had an unexpected error code.
...
Make sure your credentials are correct.
...
101
...
Unexpected status code when fetching ThreatQuotient client_id: [...]
...
The collector is having issues connecting to the ThreatQ instance.
...
Make sure you have properly configured the api_base_url setting and that you can access the {api_base_url}/assets/js/config.js URL.
...
102
...
Cannot parse client_id from ThreatQuotient server
...
The collector was expecting to find the Client’s ID, but could not find it. This is likely because the ThreatQ has been upgraded and the collector does not support it.
...
This needs to be troubleshooted by the colllector’s developers.
...
ApiError
...
400
...
Unexpected status code when fetching ThreatQuotient events: [...]
...
This error happens when the collector tries to fetch the ThreatQ events from its REST API.
...
In this error you will find the HTTP error code as long as the response’s text. This information should be enough to understand why is the error happening. Otherwise, please contact support.
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
...
| title | Verify collector operations |
|---|
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
| Code Block |
|---|
INFO MainThread -> (CollectorMultithreadingQueue) standard_queue_multithreading -> max_size_in_messages: 10000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
WARNING MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
WARNING MainThread -> [OUTPUT] OutputLookupSenders -> <threshold_for_using_gzip_in_transport_layer> setting has been modified from 1.1 to 1.0 due to this configuration increases the Lookup sender performance.
WARNING MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-us.devo.io
INFO MainThread -> [OUTPUT] OutputMultithreadingController(threatquotient_collector) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 600 seconds)
INFO MainThread -> [OUTPUT] DevoSenderManager(standard_senders,manager,devo_1)(devo_1) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO MainThread -> [OUTPUT] DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 600 seconds)
INFO MainThread -> [OUTPUT] DevoSenderManager(lookup_senders,manager,devo_1)(devo_1) -> Starting thread
INFO MainThread -> InitVariables Started
INFO MainThread -> start_time_value initialized
INFO MainThread -> verify_host_ssl_cert initialized
INFO MainThread -> event_fetch_limit_in_items initialized
INFO MainThread -> InitVariables Terminated
INFO MainThread -> [INPUT] InputMultithreadingController(threatquotient_collector) - Starting thread (executing_period=300s)
INFO MainThread -> [INPUT] InputThread(threatquotient_collector,threatquotient_data_puller#111) - Starting thread (execution_period=600s)
INFO MainThread -> [INPUT] ServiceThread(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread (execution_period=600s)
INFO MainThread -> [SETUP] ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
INFO MainThread -> [INPUT] ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread |
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
| Code Block |
|---|
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds) |
| Info |
|---|
By default, these information traces will be displayed every 10 minutes. |
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:
...
Sender services
...
Description
...
internal_senders
...
In charge of delivering internal metrics to Devo such as logging traces or metrics.
...
standard_senders
...
In charge of delivering pulled events to Devo.
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
...
Logging trace
...
Description
...
Number of available senders: 1
...
Displays the number of concurrent senders available for the given Sender Service.
...
sender manager internal queue size: 0
...
Displays the items available in the internal sender queue.
| Info |
|---|
This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
...
Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)
...
Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
44 events were sent to Devo since the collector started.
The last checkpoint timestamp was
2022-06-28 10:39:22.511671+00:00.21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required
0.007 secondsto be delivered.
| Info |
|---|
By default these traces will be shown every 10 minutes. |
| Expand | ||||
|---|---|---|---|---|
| ||||
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
|
| Expand | ||
|---|---|---|
| ||
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode. |
Change log for v1.x.x
...
Release
...
Released on
...
Release type
...
Details
...
Recommendations
...
v1.0.0
...
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
| Status | ||||
|---|---|---|---|---|
|
...
New features:
New space rocket.
Improvements:
45% more fuel capacity.
Bug Fixes:
The “Panic” protocol has been fixed and now works well.
Vulnerabilities Mitigation:
Critical - AlienTrojan 1345
High - AlienTrojan 154
Medium - AlienTrojan 0024
Low - AlienTrojan 5882
...
| Rw ui tabs macro | ||||||||
|---|---|---|---|---|---|---|---|---|
We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer:
Editing the JSON configuration
Replace the placeholders with your required values following the description table below: | ||||||||
Parameter | Data type | Type | Value range / Format | Details | ||||
|
|
| Minimum length: | Use this param to give a unique id to this input service.
| ||||
|
|
|
| This will make the collector generate (or not) log messages with the DEBUG level. | ||||
|
|
|
| Use this param to enable or disable the given input logic when running the collector. If the value is | ||||
|
|
| Minimum value: | Customize the maximum number of API requests per second. If not used, the default setting will be used:
| ||||
|
|
|
| Username to authenticate the service. It must belong to an existing user or the initial one created during the setup. | ||||
|
|
|
| Password to authenticate the service. It must belong to an existing user or the initial one created during the setup. | ||||
|
|
| Minimum value: | The amount (in seconds) in which the service’s collection is scheduled.
| ||||
|
|
| Any. Recommended: date format | This parameter allows you to clear the persistence of the collector and restart the download pipeline. Updating this value will produce the loss of all persisted data and current pipelines.
| ||||
|
|
| Must be a valid URL and comply one of these Regular Expressions:
| This parameter defines the
| ||||
|
|
|
| This should be enabled if the ThreatQ’s instance has a self-signed certificate. The usual installation steps do not include certificate signing, so this usually should be | ||||
|
|
| A date with format | This parameter allows you to clear the persistence of the collector and restart the download pipeline.
| ||||
|
|
| Minimum value: 1 | This parameter controls the collector’s pagination size when the events are fetched. The default value is
| ||||
Collector Docker image | SHA-256 hash | |||||||
|
| Code Block |
|---|
gunzip -c <image_file>-<version>.tgz | docker load |
| Note |
|---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace |
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
| Code Block |
|---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
| Note |
|---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.
| Code Block |
|---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:
| Code Block |
|---|
IMAGE_VERSION=<version> docker-compose up -d |
| Note |
|---|
Replace |
| Rw tab | ||
|---|---|---|
|
We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Events service
...
| title | Verify data collection |
|---|
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
...
Component
...
Description
...
Setup
...
The setup module is in charge of authenticating the service and managing the token expiration when needed.
...
Puller
...
The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.
Setup output
A successful run has the following output messages for the setup module:
| Code Block |
|---|
INFO MainThread -> [SETUP] ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Puller Setup Started
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> We do not have a token. Getting a new one from the server.
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Attempting to get OAuth2 token from ThreatQuotient server....
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Attempting to get from client_id ThreatQuotient server....
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Successfully received a client_id token from (...)/assets/js/config.js
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Successfully received JWT token from (...) which expires in 3599 seconds
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Puller Setup Terminated
INFO ThreatQuotientDataPullerSetup(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Setup for module "ThreatQuotientDataPuller" has been successfully executed |
Puller output
A successful initial run has the following output messages for the puller module:
| Info |
|---|
Note that the |
| Code Block |
|---|
INFO MainThread -> [INPUT] ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) - Starting thread
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Waiting until setup will be executed
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> PrePull Started
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> PrePull terminated
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Starting data collection every 5 seconds
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Pull Started. Retrieving timestamp: 2022-06-28 13:00:59.276966+00:00
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started getting events from ThreatQuotient
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started getting events from ThreatQuotient
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Started sending events to Devo
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656421259.276966): Number of requests performed: 2; Number of events received: 1; Number of duplicated events filtered out: 0; Number of events generated and sent: 2 (from 1 unflattened events); Average of events per second: 4.179186813829765.
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> last_fetched_event_id and last_update_time saved in state: {'last_polled_time': 1656342870.739774, 'reset_persistence_auth': '', 'all_events_ids': [17653]}
WARNING ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Last polled time saved in state: {'last_polled_time': 1656421259.276966, 'reset_persistence_auth': '', 'all_events_ids': [17653]}
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Pull terminated
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Data collection completed. Elapsed time: 0.483 seconds. Waiting for 4.517 second(s) until the next one |
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
| Code Block |
|---|
INFO ThreatQuotientDataPuller(threatquotient_collector,threatquotient_data_puller#111,events#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1655983326.290848): Number of requests performed: 2; Number of events received: 52; Number of duplicated events filtered out: 0; Number of events generated and sent: 52 (from 52 unflattened events); Average of events per second: 92.99414315733. |
| Info |
|---|
The value |
| Expand | ||
|---|---|---|
| ||
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
|
...
| title | Troubleshooting |
|---|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector:
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in 
Editing the config.yaml file
Replace the placeholders with your required values following the description table below:
Download the Docker imageThe collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Use the following command to add the Docker image to the system:
The Docker image can be deployed on the following services: DockerExecute the following command on the root directory
Docker ComposeThe following Docker Compose file can be used to execute the Docker container. It must be created in the
To run the container using docker-compose, execute the following command from the
|
Collector services detail
This section is intended to explain how to proceed with specific actions for services.
Services (Services)
Internal Process and Deduplication Method
The collector deduplicates by the Ids pulled and stored.
Event Deduplication
Overview
The api is queried by time intervals after pulling the ids are checked against and stored and non-duplicate events are sent to Devo
Devo Categorization and Destination
All services are tagged by the service they are pulled by.
Setup/Puller Output
| Code Block |
|---|
2024-04-02T12:36:10.881042712Z 2024-04-02T12:36:10.880 INFO InputProcess::MainThread -> InputThread(kiteworks,45635) - Starting thread (execution_period=300s)
2024-04-02T12:36:10.900848871Z 2024-04-02T12:36:10.900 INFO InputProcess::MainThread -> ServiceThread(kiteworks,45635,admin,predefined) - Starting thread (execution_period=300s)
2024-04-02T12:36:10.901635871Z 2024-04-02T12:36:10.901 INFO InputProcess::MainThread -> ManagementPullerSetup(kiteworks-collector,kiteworks#45635,admin#predefined) -> Starting thread
2024-04-02T12:36:10.902970384Z 2024-04-02T12:36:10.902 INFO InputProcess::MainThread -> ManagementPuller(kiteworks,45635,admin,predefined) - Starting thread
2024-04-02T12:36:10.903841384Z 2024-04-02T12:36:10.903 WARNING InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Waiting until setup will be executed
2024-04-02T12:36:10.910390935Z 2024-04-02T12:36:10.909 WARNING InputProcess::ManagementPullerSetup(kiteworks-collector,kiteworks#45635,admin#predefined) -> The token/header/authentication has not been created yet
2024-04-02T12:36:10.912045728Z 2024-04-02T12:36:10.911 INFO InputProcess::ManagementPullerSetup(kiteworks-collector,kiteworks#45635,admin#predefined) -> using base url: https://manage.office.com
2024-04-02T12:36:11.221983503Z 2024-04-02T12:36:11.221 INFO InputProcess::ManagementPullerSetup(kiteworks-collector,kiteworks#45635,admin#predefined) -> Setup for module <ManagementPuller> has been successfully executed
2024-04-02T12:36:11.906707525Z 2024-04-02T12:36:11.905 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> ManagementPuller(kiteworks,45635,admin,predefined) Starting the execution of pre_pull()
2024-04-02T12:36:11.907795456Z 2024-04-02T12:36:11.906 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Reading persisted data
2024-04-02T12:36:11.910462424Z 2024-04-02T12:36:11.909 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Data retrieved from the persistence: {'@persistence_version': 1, 'start_time_in_utc': None, 'last_event_time_in_utc': '2024-04-02 12:35:07'}
2024-04-02T12:36:11.911358075Z 2024-04-02T12:36:11.910 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Start time not found in config, using 2024-04-02 12:35:11
2024-04-02T12:36:11.912847398Z 2024-04-02T12:36:11.911 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Running the persistence upgrade steps
2024-04-02T12:36:11.915154717Z 2024-04-02T12:36:11.913 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Running the persistence corrections steps
2024-04-02T12:36:11.916748235Z 2024-04-02T12:36:11.915 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Running the persistence corrections steps
2024-04-02T12:36:11.918276116Z 2024-04-02T12:36:11.917 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> No changes were detected in the persistence
2024-04-02T12:36:11.919248467Z 2024-04-02T12:36:11.918 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> ManagementPuller(kiteworks,45635,admin,predefined) Finalizing the execution of pre_pull()
2024-04-02T12:36:11.920446419Z 2024-04-02T12:36:11.919 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Starting data collection every 60 seconds
2024-04-02T12:36:11.924162570Z 2024-04-02T12:36:11.923 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Pull Started
2024-04-02T12:36:12.045307400Z 2024-04-02T12:36:12.044 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Found 1 removed 0
2024-04-02T12:36:12.221770395Z 2024-04-02T12:36:12.221 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1712061371905):Number of requests made: 1; Number of events received: 30; Number of duplicated events filtered out: 0; Number of events generated and sent: 30; Average of events per second: 101.027.
2024-04-02T12:36:12.222243522Z 2024-04-02T12:36:12.222 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1712061371905):Number of requests made: 1; Number of events received: 30; Number of duplicated events filtered out: 0; Number of events generated and sent: 30; Average of events per second: 100.751.
2024-04-02T12:36:12.222631040Z 2024-04-02T12:36:12.222 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> The data is up to date!
2024-04-02T12:36:12.223216005Z 2024-04-02T12:36:12.223 INFO InputProcess::ManagementPuller(kiteworks,45635,admin,predefined) -> Data collection completed. Elapsed time: 0.318 seconds. Waiting for 59.682 second(s) until the next one```
|
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
start_time_in_utc parameterto a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Collector operations
Verify collector operations
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
| Code Block |
|---|
2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Loading configuration using the following files: {"full_config": "config-test-local.yaml", "job_config_loc": null, "collector_config_loc": null}
2023-01-10T15:22:57.146 INFO MainProcess::MainThread -> Using the default location for "job_config_loc" file: "/etc/devo/job/job_config.json"
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> "\etc\devo\job" does not exists
2023-01-10T15:22:57.147 INFO MainProcess::MainThread -> Using the default location for "collector_config_loc" file: "/etc/devo/collector/collector_config.json"
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> "\etc\devo\collector" does not exists
2023-01-10T15:22:57.148 INFO MainProcess::MainThread -> Results of validation of config files parameters: {"config": "C:\git\collectors2\devo-collector-<name>\config\config.yaml", "config_validated": True, "job_config_loc": "/etc/devo/job/job_config.json", "job_config_loc_default": True, "job_config_loc_validated": False, "collector_config_loc": "/etc/devo/collector/collector_config.json", "collector_config_loc_default": True, "collector_config_loc_validated": False}
2023-01-10T15:22:57.171 WARNING MainProcess::MainThread -> [WARNING] Illegal global setting has been ignored -> multiprocessing: False |
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method. A successful run has the following output messages for the initializer module:
| Code Block |
|---|
2023-01-10T15:23:00.788 INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2023-01-10T15:23:00.789 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2023-01-10T15:23:00.790 INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2023-01-10T15:23:00.842 INFO OutputProcess::MainThread -> global_status: {"output_process": {"process_id": 18804, "process_status": "running", "thread_counter": 21, "thread_names": ["MainThread", "pydevd.Writer", "pydevd.Reader", "pydevd.CommandThread", "pydevd.CheckAliveThread", "DevoSender(standard_senders,devo_sender_0)", "DevoSenderManagerMonitor(standard_senders,devo_1)", "DevoSenderManager(standard_senders,manager,devo_1)", "OutputStandardConsumer(standard_senders_consumer_0)", |
Sender services
The Integrations Factory Collector SDK has 3 different sender services depending on the event type to deliver (internal, standard, and lookup). This collector uses the following Sender Services:
Logging trace | Description |
|---|---|
Number of available senders: 1 | Displays the number of concurrent senders available for the given Sender Service. |
Sender manager internal queue size: 0 | Displays the items available in the internal sender queue. This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds) | Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained:
|
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace | Description |
|---|---|
Number of available senders: 1 | Displays the number of concurrent senders available for the given Sender Service |
Sender manager internal queue size: 0 | Displays the items available in the internal sender queue. |
Standard - Total number of messages sent: 57, messages sent since "2023-01-10 16:09:16.116750+00:00": 0 (elapsed 0.000 seconds | Displays the number of events from the last time the collector executed the pull logic. Following the given example, the following conclusions can be obtained:
|
Check memory usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
| Code Block |
|---|
INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB ->
410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB ->
705.28MiB) |
Change log
Release | Released on | Release type | Details | Recommendations | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
| New collector |
|