/
Office 365 Exchange Message Tracing collector

Office 365 Exchange Message Tracing collector

Configuration requirements

To run this collector, there are some configurations detailed below that you need to consider.

Configuration

Details

Configuration

Details

Microsot Azure account

It is a requirement to have an account so you can run this collector.

Microsoft 365 account

It is a requirement to have an account so you can run this collector.

More information

Refer to the Vendor setup section to know more about these configurations.

Overview

Office 365 is a widely adopted productivity suite from Microsoft. The suite includes Microsoft Exchange Online collaboration service for email, calendars, contacts, and tasks. Message tracing is one of the basic tools used by administrators to monitor email flow.

Some email information is stored in logs and is available for administrative purposes. The administrator will be able to see sent and received emails even though the user has deleted or purged them. With message tracing, you can’t see the content of the messages but you can get very useful information such as:

  • Sender and recipient.

  • Send and received dated.

  • Subject and size of the email.

  • Status and details of events. There are seven possible values in the delivery status field: delivered, failed, pending, expanded, quarantined, filtered as spam, and unknown.

  • IP address used to send the message.

  • Message ID: This is a unique number that identifies the message. If a message is sent to more than one recipient it will display once for every recipient, but all will have the same Message ID.

Message tracking logs

You won’t be able to edit the messages in Office 365 since they are not simple text files. To do that you can use message tracking logs in on-prem Exchange that allow you to edit them directly since they are simple text files.

Devo collector features

Feature

Details

Feature

Details

Allow parallel downloading (multipod)

Not allowed

Running environments

Collector server

On-premise

Populated Devo events

Table

Flattening preprocessing

No

Data sources

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Message Reports

Reports for every email sent and received through the Exchange Online organization

https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace

office365_exchange_message_service

cloud.office365.messagetracing

2.0.0

For more information on how the events are parsed, visit our page.

Vendor setup

There are some minimal requirements to enable this collector:

  • Microsoft Azure account: The account must be able to register applications and assign roles.

  • Microsoft Office 365 account: The account must have administrative permissions in the Office 365 organization.

There are some configurations you need to do to enable the collector.

  1. Go to Microsoft Azure and log in with your account.

  2. Navigate the App registration blade (you can search for it in the search toolbar).

  3. Click on the + New Registration button.

  4. Give the application a name. If needed change the Supported account types.

  5. Click the Register button.

  1. On the Overview page, search for your application in the list.

  2. Under the Essentials section, copy the Application (client) ID and Directory (tenant) ID values.

  3. On the left side menu, navigate to the Certificates & secrets page.

  4. Click on + New client secret button.

  5. Set a description and an expiration for the secret.

  6. Click on the Add button.

  7. Copy its Value field.

  1. On the left side panel of your previously registered application, navigate to the API permissions page.

  2. Click the + Add a permission button.

  3. On the right side modal, navigate the APIs my organization uses tab.

  4. Select Office 365 Exchange Online entry from the list - you can use the search bar tool to find it -.

  5. Click the Application permissions button.

  6. Under the ReportingWebService section, mark the ReportingWebService.Read.All permission.

  7. Click the Add permissions button.

  8. Click ✓ Grant admin consent for {your_domain} button and then on the Yes button on the appeared modal.

  9. To verify that the permission was correctly added, execute the following cURL command replacing (or assigning) the $CLIENT_ID, $CLIENT_SECRET, and $TENANT_ID shell variables with the credentials you obtained in the previous instructions:

    $ curl -d "grant_type=client_credentials&client_id=$CLIENT_ID&client_secret=$CLIENT_SECRET&resource=https://outlook.office365.com" -X POST https://login.windows.net/$TENANTID/oauth2/token

    A JSON response will be received containing an "access_token" field. By copying its value and pasting it on the jwt.ms: Welcome! web page, the token will appear decoded. Inside, there must be a "roles" field whose value is an array containing the ReportingWebService.Read.All permission. Otherwise, the permissions have not been correctly added.

  1. Navigate the Azure Active Directory blade (you can search for it in the search bar on the top).

  2. On the left-side menu, navigate to the Roles and administrators page.

  3. In the list, search for either the Global Reader or Security Reader role (both are valid) and click on it to select it.

  4. Click the + Add assignments button.

  5. On the right-side modal, use the search bar to find the application you created. Search by either its name (make sure you write the whole name, not only parts of it) or its Client ID in case you cannot find it. Once it appears on the list, select it.

  6. Click the Add button.

  7. To verify that the role has been correctly assigned, execute the following cURL command replacing (or assigning) the $REPORTING_TOKEN shell variable with the token you obtained in the previous instructions:

    $ curl -i -H "Authorization: Bearer $REPORTING_TOKEN" https://reports.office365.com/ecp/ReportingWebService/Reporting.svc

    This response will include the HTTP Response code and headers (note the -i parameter). You should expect to get an HTTP 200 OK response with an XML containing at least the <atom:title>MessageTrace</atom:title> tag. Other responses will be incorrect; here are some examples and their explanation:

    1. 401 - Unauthorized: Access is denied due to invalid credentials.: your token is either invalid or outdated. Review the previous instructions to obtain a working token.

    2. No permission to access the report for the organization.: The role has not been properly assigned. Review these instructions to do it.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

Setting

Details

tenant_id

This is the Tenant ID you copied during the Obtain the client credentials step.

client_id

This is the Application (client) ID you copied during the Obtain the client credentials step.

client_secret

This is the created secret’s Value field you copied during the Obtain the client credentials step.

Accepted authentication methods

Authentication method

Username

Password

Tenant ID

Client ID

Client Secret

Authentication method

Username

Password

Tenant ID

Client ID

Client Secret

Modern OAuth2 Auth

 

 

REQUIRED

REQUIRED

REQUIRED

Basic User/Password Auth

DEPRECATED

DEPRECATED

 

 

 

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Events service

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::MainThread -> Office365ExchangeMessageTracePullerSetup(unknown,office365_exchange_message#<short_unique_id>,office365_exchange_message_service#predefined) -> Starting thread INFO InputProcess::Office365ExchangeMessageTracePullerSetup(unknown,office365_exchange_message#<short_unique_id>,office365_exchange_message_service#predefined) -> Puller Setup started INFO InputProcess::Office365ExchangeMessageTracePullerSetup(unknown,office365_exchange_message#<short_unique_id>,office365_exchange_message_service#predefined) -> We do not have a token. Getting a new one from the server. INFO InputProcess::Office365ExchangeMessageTracePullerSetup(unknown,office365_exchange_message#<short_unique_id>,office365_exchange_message_service#predefined) -> Attempting to get OAuth2 token from O365 Message Tracing server.... INFO InputProcess::Office365ExchangeMessageTracePullerSetup(unknown,office365_exchange_message#<short_unique_id>,office365_exchange_message_service#predefined) -> Successfully received JWT token from https://login.microsoftonline.com/{your-tenant-id}/oauth2/token which expires in 3599 seconds INFO InputProcess::Office365ExchangeMessageTracePullerSetup(unknown,office365_exchange_message#<short_unique_id>,office365_exchange_message_service#predefined) -> Puller Setup terminated INFO InputProcess::Office365ExchangeMessageTracePullerSetup(unknown,office365_exchange_message#<short_unique_id>,office365_exchange_message_service#predefined) -> Setup for module <Office365ExchangeMessageTracePuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Change log for v1.x.x

Release

Released on

Release type

Details

Recommendations

Release

Released on

Release type

Details

Recommendations

v2.0.0

Dec 23, 2022

NEW FEATURE
IMPROVEMENT

New features:

  1. New OAuth2 authentication method; basic authentication deprecated.

  2. New max_historic_date_cutoff_days_back and max_historic_date_margins_second parameters to customize historic pulling.

Improvements:

  1. Puller setup is now handled in its own class.

Recommended version