Page Comparison

...

Rw ui tabs macro

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the collector:

Code Block

<any_directory>
└── devo-collectors/
    └── <product_name>/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        ├── state/
        └── config/ 
            └── config.yaml

Note
Replace `<product_name>` with the proper value.

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/. Learn more about security credentials in Devo here.

Note
Replace `<product_name>` with the proper value.

Editing the config.yaml file

Code Block

globals:
  debug: <enable_debug_logs>
  id: not_used
  name: o365_emt_collector
  persistence:
    type: filesystem
    config:
      directory_name: state

outputs:
  devo_1:
    type: devo_platform
    config:
      address: collector-us.devo.io
#      address: collector-eu.devo.io
      port: 443
      type: SSL
      chain: chain.crt
      cert: <devo_domain>.crt
      key: <devo_domain>.key
inputs:
  office365_exchange_message:
    id: <short_unique_id>
    enabled: <input_status>
    requests_per_second: <requests_per_second_value>
    max_historic_date_cutoff_days_back: <max_historic_date_cutoff_days_back_value>
    max_historic_date_margin_seconds: <max_historic_date_margin_seconds_value>
    credentials:
      tenant_id: <tenant_id_value>
      client_id: <client_id_value>
      client_secret: <client_secret_value>
    services:
      office365_exchange_message_service:
        request_period_in_seconds: <request_period_in_seconds_value>
        reset_persistence_auth: <reset_persistence_auth_value>
        override_time_delay_in_seconds: <override_time_delay_in_seconds_value>

Info
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the `services` object.

Replace the placeholders with your required values following the description table below:

Parameter

Data type

Type

Value range / Format

Details

<enable_debug_logs>

bool

Mandatory

false / true

This will make the collector generate (or not) log messages with the DEBUG level.

<short_unique_id>

int

Mandatory

YMMDD

Use this param to give a unique id to this input service.

Note
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision.

<input_status>

bool

Mandatory

false / true

Use this param to enable or disable the given input logic when running the collector. If the value is true, the input will be run. If the value is false, it will be ignored.

<requests_per_second_value>

int

Optional

Minimum value: 1

Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000 requests/sec.

Info
This parameter should be removed if it is not used.

<max_historic_date_cutoff_days_back_value>

int

Optional

Minimum value: 1

Maximum value: 1,000,000

When the collector is initially run, it will perform a historical pulling from the current date minus the value of this parameter. The default value is 10 (the API does not allow querying more than 10 days back).

Info
This parameter should be removed if it is not used.

<max_historic_date_margin_seconds_value>

int

Optional

Minimum value: 1

Maximum value: 86,400,000,000

It happens that between when this date is calculated and when it is used, the difference might be larger than 10 days, so the request will fail. To avoid it, we establish a margin that can be configured using this parameter. Its default value is 60.

Info
This parameter should be removed if it is not used.

<tenant_id_value>

str

Mandatory

Minimum length: 1

This is the Tenant ID you copied during the Obtain the client credentials step from the Vendor setup section.

<client_id_value>

str

Mandatory

Minimum length: 1

This is the Application (client) ID you copied during the Obtain the client credentials step from the Vendor setup section.

<client_secret_value>

str

Mandatory

Minimum length: 1

This is the created secret’s Value field you copied during the Obtain the client credentials step from the Vendor setup section.

<request_period_in_seconds_value>

int

Optional

Minimum value: 1

The amount (in seconds) in which the service’s collection is scheduled.

Info
This parameter should be removed if it is not used.

<reset_persistence_auth_value>

str

Optional

Date format: YYYY-MM-DD

This parameter allows you to clear the persistence of the collector and restart the download pipeline.

Note
Updating this value will produce the lost of all persisted data and current pipelines.

Info
This parameter should be removed if it is not used.

<override_delay_in_seconds_value>

int

Optional

Minimum value: 1
Maximum value: 172,800

Because it can take up to 24 hours for an event to be available through the API, this collector requests data with a 25 hour delay. This parameter allows you to customize this delay.

Note
A value less than 25 hours could cause the loss of events in large infrastructures.

Info
This parameter should be removed if it is not used.

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-office365_exchange_message_tracing-2.1.1	`3bbaa2f308c7023d7ec42e11390c620ff18b3e3f98957effa7e691bb3bb591327344e30169c1c438a5a9b431a4c37733d5104d9abe01ef4d6061640938630b17`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c <image_file>-<version>.tgz \| docker load

Note
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace `<image_file>` and `<version>` with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/

Code Block

docker run 
--name collector-<product_name> 
--volume $PWD/certs:/devo-collector/certs 
--volume $PWD/config:/devo-collector/config 
--volume $PWD/state:/devo-collector/state 
--env CONFIG_FILE=config.yaml 
--rm 
--interactive 
--tty 
<image_name>:<version>

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/ directory.

Code Block

version: '3'
services:
  collector-<product_name>:
    image: <image_name>:${IMAGE_VERSION:-latest}
    container_name: collector-<product_name>
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./credentials:/devo-collector/credentials
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<product_name>`, `<image_name>` and `<version>` with the proper values.

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

...

Expand

title	Enable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Change log

...

Release
Released on
Release type
Details
Recommendations
`v2.2.0`
11 Sep 2023
Status
colour Green
title IMPROVEMENT
Improvements:
Upgraded DCSDK from 1.9.0 to 1.9.2
Store lookup instances into DevoSender to avoid creation of new instances for the same lookup
Ensure service_config is a dict into templates
Upgraded some internal dependencies
Updated collector definitions with new rate limits and reverted changes to older version of collector
`Recommended version`
`v2.1.0`
21 Jul 2023
Status
colour Red
title BUG FIXING
Status
colour Green
title IMPROVEMENT
Improvements:
Upgraded DCSDK from 1.3.0 to 1.8.0
Bug fixing:
Processing of special characters that the collector was not capable of doing. It has been fixed by adding the processing of non-ascii characters
`Update`
`v2.0.0`
23 Dec 2022
Status
colour Purple
title NEW FEATURE

Status
colour Green
title IMPROVEMENT
New features:
New OAuth2 authentication method; basic authentication deprecated.
New `max_historic_date_cutoff_days_back` and `max_historic_date_margins_second` parameters to customize historic pulling.
Improvements:
Puller setup is now handled in its own class.
`Recommended version`
`-`

Versions Compared

Old Version 12

New Version 13

Key

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Change log