We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer: In the Collector Server GUI, access to the domain in which you want this instance to be created in, click on Add Collector and search for “Mimecast Collector - Integrations Factory”, then click on the result. In the Version field, select the latest value. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain). In the Parameters section, establish the Collector Parameters as follows below:
Editing the JSON configuration Code Block |
---|
{
"global_overrides": {
"debug": false
},
"inputs": {
"mimecast_input": {
"id": "<short_unique_identifier>",
"enabled": true,
"base_url": "your_base_url",
"auth_url": "your_auth_url",
"pageSize": "<page_size_value>",
"autoconfig": {
"refresh_interval_in_seconds": "refresh_interval_value",
"creation_timeout_in_second": "creation_timeout_value"
},
"credentials": {
"client_id": "your_client_id",
"client_secret": "your_client_secret",
"app_id": "your_app_id",
"app_key": "your_app_key",
"access_key": "your_access_key",
"secret_key": "your_secret_key"
},
"services": {
"service_mimecast_client_api": {
initial_lookback_period: 1d"last_configuration_timestamp": "last_configuration_timestamp_value",
- "endpoints_8": [
name:{
messageholdlist initial_lookback_period"endpoints_1": 1d{
- endpoints_9: "name": "audit",
name: messageholdsummary "initial_lookback_period": "1d"
- endpoints_10: }
name: dashboard},
{
initial_lookback_period: 1d mimecast_siem_input: id: 2 enabled"endpoints_2": false{
requests_per_second: 5 base_url: your_base_url auth_url: your_auth_url"name": "attachments",
pageSize: 10 autoconfig: refresh"initial_interval_in_seconds: 60lookback_period": "1d"
# Runs the setup every}
x seconds (default 600) creation_timeout_in_second: 60 },
# Set up the setup timeout (default{
60) credentials: client"endpoints_id3": <client_id_value> {
client_secret: <client_secret_value> services:"name": "impersonation",
service_mimecast_siem_client_api: last"initial_configurationlookback_timestamp: 2021-12-02T13:10:00Z # change this if you want to get your state changed!period": "1d"
}
},
{
"endpoints_4": {
siem:"name": "url",
"initial_lookback_period": "1d
|
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Replace the placeholders with your required values following the description table below: |
Parameter | Data type | Type | Value range/ Format | Details |
---|
debug
| bool
| Mandatory
| false / true | This will make the collector generate (or not) log messages with the DEBUG level. |
id
| str
| Mandatory | Minimum length: 1 Maximum length: 5 | Alphanumeric identifier. |
enabled
| bool
| Mandatory | true /false
| Enables or disables the input. |
base_url
| str
| Mandatory | | The region related to the API credentials, the string value has to be one of the valid URL’s in Global Base URLs | Mimecast. |
credentials
| dictionary
| Mandatory | Code Block |
client_id: <client_id_value>"
}
},
{
"endpoints_5": {
"name": "search",
"initial_lookback_period": "1d"
}
},
{
"endpoints_6": {
"name": "view",
"initial_lookback_period": "1d"
}
},
{
|
clientsecret: <client_secret_value>Credentials to use the API. | endpoints
| list
| Mandatory | Minimum length: 1
Posible values:
Code Block |
- endpoints_1:name:auditinitial_lookback_period: 1d
- endpoints_2:name:attachments "initial_lookback_period": "1d"
|
-endpoints_3:name:impersonationinitial_lookback_period:1d-endpoints_4:name:urlinitial_lookback_period: 1d
- endpoints_5:name:searchinitial_lookback_period: 1d
- endpoints_6:name:viewinitial_lookback_period:1d-7 name:threatfeedinitial_lookback_period:1d-endpoints_8:
initial_lookback_period:1d-endpoints_9:name:messageholdsummary "initial_lookback_period": "1d"
|
-endpoints_10:name:dashboardinitial_lookback_period: 1d
An array with at least one endpoint, the collector will pull from the selected endpoints. | last_configuration_timestamp
| str
| Mandatory | Date following the next format: yyyy-mm-ddThh:mm:ss.000Z
| Change this value to a date after the initial configuration to reset the state of the collector. |
initial_lookback_period
| str
| Mandatory | Number of days, Example: | This value will be subtracted from the current date to execute all queries in that range if no state is detected (Initial execution for example). This value only has an effect for mimecast_input , mimecast_siem_input always pull from the last 7 days. |
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Use the following command to add the Docker image to the system:
Code Block |
---|
gunzip -c <image_file>-<version>.tgz | docker load |
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file>
and <version>
with a proper value.
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Code Block |
---|
docker run \
--name <YOUR_COLLECTOR_NAME>\
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=<YOUR-CONFIG-FILE>.yaml \
--rm -it <YOUR_DEVO_IMAGE_NAME>:<version> |
Note |
---|
Replace <product_name> , <image_name> and <version> with the proper values. |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/
directory.
Code Block |
---|
version: '3'
services:
<YOUR_COLLECTOR_NAME>:
image: <YOUR_DEVO_IMAGE_NAME>:${IMAGE_VERSION:-latest}
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-<YOUR-CONFIG-FILE>.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/
directory:
Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace <product_name> , <image_name> and <version> with the proper values. |
We use a piece of software called Collector Server to host and manage all our available collectors.
To enable the collector for a customer:
In the Collector Server GUI, access to the domain in which you want this instance to be created in, click on Add Collector and search for “Mimecast Collector - Integrations Factory”, then click on the result.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the Parameters section, establish the Collector Parameters as follows below:
Editing the JSON configuration
Code Block |
---|
{
"mimecast_input": {}
},
{
"endpoints_9": {
"name": "messageholdsummary",
"initial_lookback_period": "1d"
}
},
{
"endpoints_10": {
"name": "dashboard",
"initial_lookback_period": "1d"
}
}
]
}
}
},
"mimecast_siem_input": {
"id": "<short_unique_identifier>",
"enabled": true,
"requests_per_second": "requests_per_second_value",
"base_url": "your_base_url",
"auth_url": "your_auth_url",
"idpageSize": "<shortpage_uniquesize_identifier>value",
"enabledautoconfig": {
true, "base_urlrefresh_interval_in_seconds": "yourrefresh_baseinterval_urlvalue",
"auth_urlcreation_timeout_in_second": "yourcreation_authtimeout_urlvalue",
"pageSize": "<page_size_value>"},
"autoconfigcredentials": {
"refresh_interval_in_secondsclient_id": "refreshyour_intervalclient_valueid",
"creation_timeout_in_secondclient_secret": "creationyour_timeoutclient_value"
secret",
}, "credentialsapp_id": {"your_app_id",
"clientapp_idkey": "your_clientapp_idkey",
"clientaccess_secretkey": "your_clientaccess_secret"
key",
}, "secret_key": "your_secret_key"
},
"services": {
"service_mimecast_siem_client_api": {
"last_configuration_timestamp": "last_configuration_timestamp_value",
"endpoints": [
{
"endpoints_1siem": {
"nameinitial_lookback_period": "audit0d",
"initialpage_lookback_periodtoken": "1d<page_token>"
}
},
}
{ }
}
"endpoints_2": {
"name": "attachments",
"initial_lookback_period": "1d"
}}
} |
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Please replace the placeholders with real world values following the description table below:
Parameter | Data Type | Type | Value Range/ Format | Details |
---|
id
| str
| Mandatory | Minimum length: 1 Maximum length: 5 | Alphanumeric identifier. |
enabled
| bool
| Mandatory | true /false
| Enables or disables the input. |
base_url
| str
| Mandatory | For v2 API
https://api.services.mimecast.com For v1 API, see: Global Base URLs | Base url for all the APIs |
auth_url
| str
| Mandatory | For v2 API:
"https://api.services.mimecast.com/oauth/token" For v1: Delete this parameter | Auth url to generated auth token. |
credentials
| dictionary
| Mandatory | For v2 API: Code Block |
---|
"credentials": {
|
|
}"client_id": "your_client_id",
|
|
{
"client_secret": "your_client_secret"
}, |
For v1: Code Block |
---|
"credentials": {
" |
|
endpoints3{
nameimpersonation",
initiallookback_period1d }
},
"secret_key": "your_secret_key"
} |
| Credentials to use the API. |
endpoints
| list
| Mandatory | Minimum length: 1 Posible values: |
4
url
"initial_lookback_period": "1d" |
|
5
search "initial_lookback_period": "1d"
|
|
6
view
"initial_lookback_period": "1d" |
|
7
threatfeed
"initial_lookback_period": "1d" |
|
8
messageholdlist
"initial_lookback_period": "1d" |
|
9
messageholdsummary
"initial_lookback_period": "1d" |
|
10
dashboard
"initial_lookback_period": "1d"
}
|
|
}}]}
}},
mimecast_siem_input": {
"id<short_unique_identifier>"enabled":true,requestspersecondrequests_per_second_value","base_url":"your_base_url","auth_url":"your_auth_url","pageSize": "page_size_value""autoconfig":{"refresh_interval_in_seconds": "refresh_interval_value",creation_timeout_in_secondcreation_timeout_value"},credentialsinitial_lookback_period": |
|
{"client_id":"your_client_id","client_secret":"your_client_secret"},"services":"service_mimecast_siem_client_api": {
lastconfiguration_timestamp"last_configuration_timestamp_value",
"endpoints": {siem{"dashboard",
"initial_lookback_period": "1d"
}
}
|
|
}
}
}
} Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Please replace the placeholders with real world values following the description table below:
Parameter | Data Type | Type | Value Range/ Format | Details |
---|
id
| str
| Mandatory | Minimum length: 1 Maximum length: 5 | Alphanumeric identifier. |
enabled
| bool
| Mandatory | true /false
| Enables or disables the input. |
base_url
| str
| Mandatory | | The region related to the API credentials, the string value has to be one of the valid URL’s in https://integrations.mimecast.com/documentation/api-overview/global-base-urls/. |
credentials
| dictionary
| Mandatory | Code Block |
"credentials": {
"client_id": "your_client_id", | An array with at least one endpoint, the collector will pull from the selected endpoints. |
last_configuration_timestamp
| str
| Mandatory | Date following the next format: yyyy-mm-ddThh:mm:ss.000Z
| Change this value to a date after the initial configuration to reset the state of the collector. |
initial_lookback_period
| str
| Mandatory | Number of days, Example: | This value will be subtracted from the current date to execute all queries in that range if no state is detected (Initial execution for example). This value only has an effect for mimecast_input , mimecast_siem_input always pull from the last 7 days. |
page_token
| str
| Optional | Token from app log | Advanced theme: it is possible to put a pagination token from the collector to start fetching data from a given page. |
We recommend to leave parameters not in the list with their default values.
Keep in mind that the Mimecast collector has two different inputs:
mimecast_input
mimecast_siem_input
The collector can use both inputs or just one. Each input uses different endpoints and feeds different tables in Devo. Make sure to check the credentials given to determine the inputs and endpoints to use.
Rw tab |
---|
title | On-premise collector |
---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
Code Block |
---|
<any_directory>
└── devo-collectors/
└── <product_name>/
|
"client_secret": "your_client_secret"
},Credentials to use the API. | endpoints
| list
| Mandatory | Minimum length: 1
Posible values:
Code Block |
[{"endpoints_1": {<your_domain>.key
│ └── <your_domain>.crt
|
"name":"audit", "initial_lookback_period": "1d"}
},
{
"endpoints_2": {
"name": "attachments",
"initial_lookback_period": "1d" Note |
---|
Replace <product_name> with the proper value. |
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/
. Learn more about security credentials in Devo here.
Image Added Note |
---|
Replace <product_name> with the proper value. |
Editing the config.yaml file
Code Block |
---|
globals:
debug: false
id: not_used
name: mimecast_collector
persistence:
type: filesystem
config:
directory_name: state
queue_max_size_in_messages: 1000
|
}
queue_wrap_max_size_in_messages: 100
outputs:
# devo_1:
# |
},
type: devo_platform
# config:
# |
{ address: collector-us.devo.io
# |
"endpoints_3"{"name": "impersonation",chain: chain.crt
# cert: <devo_domain>.crt
# key: |
"initial_lookback_period": "1d"
<devo_domain>.key
inputs:
mimecast_input:
id: 1
|
}enabled: true
requests_per_second: 5
|
}, base_url: your_base_url
auth_url: your_auth_url
|
{ "endpoints_4"{ refresh_interval_in_seconds: 60 # Runs |
"name": "url",
the setup every x seconds (default 600)
|
"initial_lookback_period": "1d"creation_timeout_in_second: 60 # Set up the setup |
}timeout (default 60)
credentials:
|
}, client_id: <client_id_value>
|
{
client_secret: <client_secret_value>
|
"endpoints_5": {app_id: <app_id_value>
app_key: <app_key_value>
|
"name": "search",access_key: <access_key_value>
secret_key: <secret_key_value>
|
"initial_lookback_period""1d" service_mimecast_client_api:
|
} last_configuration_timestamp: 2021-12-02T13:10:00Z # change this if you want to |
},{"6" {"""view",""""}},{ initial_lookback_period: 1d
|
"7" {"""threatfeed","""1d"}},{
initial_lookback_period: 1d
- |
"8" {"""messageholdlist",""""}},{ initial_lookback_period: 1d
|
"endpoints_9":{"""messageholdsummary","""1d"}},{
initial_lookback_period: 1d
- |
"10": {
"name": "dashboard",
"initial_lookback_period": "1d"
}
}
]An array with at least one endpoint, the collector will pull from the selected endpoints. | last_configuration_timestamp
| str
| Mandatory | Date following the next format: yyyy-mm-ddThh:mm:ss.000Z
| Change this value to a date after the initial configuration to reset the state of the collector. |
initial_lookback_period
| str
| Mandatory | Number of days, Example: | This value will be subtracted from the current date to execute all queries in that range if no state is detected (Initial execution for example). This value only has an effect for mimecast_input , mimecast_siem_input always pull from the last 7 days. |
We recommend to leave parameters not in the list with their default values.
Keep in mind that the Mimecast collector has two different inputs:
mimecast_input
mimecast_siem_input
The collector can use both inputs or just one. Each input uses different endpoints and feeds different tables in Devo. Make sure to check the credentials given to determine the inputs and endpoints to use. Using Postman is a great way to check the credentials to each input. Read more here.
Input | Endpoint | Tables |
---|
mimecast_input
| /api/audit/get-audit-events
/api/ttp/attachment/get-logs
/api/ttp/impersonation/get-logs
/api/ttp/url/get-logs
/api/archive/get-search-logs
/api/archive/get-view-logs
/api/ttp/threat-intel/get-feed
/api/gateway/get-hold-message-list
/api/gateway/get-hold-summary-list
/api/account/get-dashboard-notifications
| mail.mimecast.audit.events
mail.mimecast.ttp.attachment
mail.mimecast.ttp.impersonation
mail.mimecast.ttp.url
mail.mimecast.archive.search
mail.mimecast.archive.messageview
mail.mimecast.threat.feed
mail.mimecast.message.list
mail.mimecast.message.summary
mail.mimecast.account.dashboard
|
mimecast_siem_input
| | mail.mimecast.siem.receipt
mail.mimecast.siem.process
mail.mimecast.siem.delivery
mail.mimecast.siem.jrnl
mail.mimecast.siem.av
mail.mimecast.siem.iep
mail.mimecast.siem.impersonation
mail.mimecast.siem.spameventthread
mail.mimecast.siem.ttp
9:
name: messageholdsummary
initial_lookback_period: 1d
- endpoints_10:
name: dashboard
initial_lookback_period: 1d
mimecast_siem_input:
id: 2
enabled: false
requests_per_second: 5
base_url: your_base_url
auth_url: your_auth_url
pageSize: 10
autoconfig:
refresh_interval_in_seconds: 60 # Runs the setup every x seconds (default 600)
creation_timeout_in_second: 60 # Set up the setup timeout (default 60)
credentials:
client_id: <client_id_value>
client_secret: <client_secret_value>
app_id: <app_id_value>
app_key: <app_key_value>
access_key: <access_key_value>
secret_key: <secret_key_value>
services:
service_mimecast_siem_client_api:
last_configuration_timestamp: 2021-12-02T13:10:00Z # change this if you want to get your state changed!
endpoints:
siem:
initial_lookback_period: 0d
page_token: <page_token> |
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the services object. |
Replace the placeholders with your required values following the description table below:
Parameter | Data type | Type | Value range/ Format | Details |
---|
id
| str
| Mandatory | Minimum length: 1 Maximum length: 5 | Alphanumeric identifier. |
enabled
| bool
| Mandatory | true /false
| Enables or disables the input. |
base_url
| str
| Mandatory | For v2 API
https://api.services.mimecast.com For v1 API, see: Global Base URLs | Base url for all the APIs |
auth_url
| str
| Mandatory | For v2 API:
"https://api.services.mimecast.com/oauth/token" For v1: Delete this parameter | Auth url to generated auth token. |
credentials
| dictionary
| Mandatory | Code Block |
---|
client_id: <client_id_value>
client_secret: <client_secret_value> |
| Credentials to use the API. |
endpoints
| list
| Mandatory | Minimum length: 1 Posible values: Code Block |
---|
- endpoints_1:
name: audit
initial_lookback_period: 1d
- endpoints_2:
name: attachments
initial_lookback_period: 1d
- endpoints_3:
name: impersonation
initial_lookback_period: 1d
- endpoints_4:
name: url
initial_lookback_period: 1d
- endpoints_5:
name: search
initial_lookback_period: 1d
- endpoints_6:
name: view
initial_lookback_period: 1d
- endpoints_7:
name: threatfeed
initial_lookback_period: 1d
- endpoints_8:
name: messageholdlist
initial_lookback_period: 1d
- endpoints_9:
name: messageholdsummary
initial_lookback_period: 1d
- endpoints_10:
name: dashboard
initial_lookback_period: 1d
|
| An array with at least one endpoint, the collector will pull from the selected endpoints. |
last_configuration_timestamp
| str
| Mandatory | Date following the next format: yyyy-mm-ddThh:mm:ss.000Z
| Change this value to a date after the initial configuration to reset the state of the collector. |
initial_lookback_period
| str
| Mandatory | Number of days, Example: | This value will be subtracted from the current date to execute all queries in that range if no state is detected (Initial execution for example). This value only has an effect for mimecast_input , mimecast_siem_input always pull from the last 7 days. |
page_token
| str
| Optional | Token from app log | Advanced theme: it is possible to put a pagination token from the collector to start fetching data from a given page. |
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Use the following command to add the Docker image to the system:
Code Block |
---|
gunzip -c <image_file>-<version>.tgz | docker load |
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace <image_file>
and <version>
with a proper value.
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Code Block |
---|
docker run \
--name <YOUR_COLLECTOR_NAME>\
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=<YOUR-CONFIG-FILE>.yaml \
--rm -it <YOUR_DEVO_IMAGE_NAME>:<version> |
Note |
---|
Replace <product_name> , <image_name> and <version> with the proper values. |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/
directory.
Code Block |
---|
version: '3'
services:
<YOUR_COLLECTOR_NAME>:
image: <YOUR_DEVO_IMAGE_NAME>:${IMAGE_VERSION:-latest}
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-<YOUR-CONFIG-FILE>.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/
directory:
Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace <product_name> , <image_name> and <version> with the proper values. |