Table of Contents | ||||||
---|---|---|---|---|---|---|
|
...
This collector lets you build, deploy and scale applications, websites, and services on the same infrastructure as Google. It also provides the possibility to integrate the Google Cloud Platform (GCP) with the Devo platform making it easy to query and analyze GCP event data. You can view it in the pre-configures Activeboards or you can customize it.
Devo’s GPC GCP collector also enables to retrieve data stored in the GPC GCP via Google Cloud APIs such as audit logs, Security Command Center findings, networking, load balance, and more available via Pub/Sub into Devo to query, correlate, analyze and visualize to enable Enterprise IT and Cybersecurity teams to take the most impactful decisions at the petabyte scale.
...
Info |
---|
The diagram is only an example to the GPC GCP services. There are many more GCP services. |
The Logging service has different ways of exporting the information stored and structured in messages. In this case, it’s being used by another GPC GCP service called PubSub, basically, this service will contain a topic object that will receive a filtered set of messages from the Logging service, then the GPC GCP collector will retrieve all those messages from the topic object using a subscription (in the pull mode).
...
GCP Project: You need to have a GPC GCP Project in the console to be able to receive data.
Service account: The Service account is a Google service that allows.
GCP Pub/Sub: It is the queue from which the events will be downloaded, it is necessary to create a Topic and a Subscription.
Sink (optional): The sink is a filter to receive only the type of events that you want.
...
Here you will find the steps to filter this type of event:
Action | Steps | |||
---|---|---|---|---|
1 | Activate Security Command Center service
| In order to receive this type of event, it is necessary to have the Security Command Center service activated. Refer to the Security Command Center Quickstart video from the Google guide. | ||
2 | Setting up a new topic | Refer to the Configuration of the Loggingsection to know how to do it. | ||
3 | Setting up a Pub/Sub | Refer to the Configuration of the Loggingsection to know how to do it. | ||
4 | Setting up a sink | Refer to the Configuration of the Loggingsection to know how to do it. |
...
Note |
---|
You have to use the |
Action | Steps | |
1 | Configure Identity and Access Management (IAM) roles. | Refer to the official Google guide in which additional configurations are described. |
2 | Activate the Security Command Center API. | |
3 | Setting up a Pub/Sub topic. | |
4 | Creating a Notification configuration. |
...
Authentication method | Project ID | Base64 credentials | File credentials | Available on | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Service account with Base64. |
|
|
| |||||||||||||
2 | Service account with the file credentials. |
|
|
|
...
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Rw ui tabs macro | |||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
│ ├── <your_domain>.key
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running. StructureThe following directory structure should be created for being used when running the collector: Code Block |
We use a piece of software called Collector Server to host and manage all our available collectors. To enable the collector for a customer:
Editing the JSON configuration
Devo credentialsIn Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in
Editing the config.yaml file
Replace the placeholders with your required values following the description table below: | ||||||||||||||||
Parameter | Data Type | Type | Value Range / Format | Details |
}
} |
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the |
Please replace the placeholders with real world values following the description table below:
Use the following command to add the Docker image to the system:
Code Block |
---|
gunzip -c <image_file>-<version>.tgz | docker load |
Note |
---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace |
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Code Block |
---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
Note |
---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/
directory.
Parameter | Data type | Type | Value range / Format | Details | ||
|
|
| Minimum length: 1 | Use this param to give an unique id to this input service.
| ||
|
|
|
| Use this param to enable or disable the given input logic when running the collector. If the value is | ||
|
|
| Minimum value: 1 | Customize the maximum number of API requests per second. If not used, the default setting will be used: This parameter should be removed if it is not used. | ||
|
|
| Minimum length: 1 | Use this param to |
Note |
---|
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. |
input_status
bool
Mandatory
false
/ true
Use this param to enable or disable the given input logic when running the collector. If the value is true
, the input will be run. If the value is false
, it will be ignored.
requests_per_second
int
Optional
Minimum value: 1
Customize the maximum number of API requests per second. If not used, the default setting will be used: 100000
requests/sec.
assign a custom name for identifying the environment of the infrastructure.
| ||||||
|
|
| Minimum length: 1 | The name of the GCP project. The Configuration of the Logging service section guide shows how to get this value.
| ||
|
|
| Minimum length: 1 | The file name with the account credentials. The Configuration of the Logging service section guide shows how to get this value.
This parameter should be removed if it is not used. |
|
int
Mandatory
Minimum length: 1
Maximum length: 5
Use this param to assign a custom name for identifying the environment of the infrastructure.
Info |
---|
This |
|
|
Mandatory
| Minimum length: 1 | The |
Service Account credentials in base64. The Configuration of the Logging service section guide shows how to get this value.
|
project_id
will be added to all events ingested by this collector in the field @devo_project_id
and helps you to filter based on this entity.filename_value
str
Optional
Minimum length: 1
The file name with the account credentials. The Configuration of the Logging service section guide shows how to get this value.
filename_value
and file_
|
This parameter should be removed if it is not used. |
file_content_base64_value
|
|
Optional
| Minimum length: 1 | The |
ID of the Pub/sub subscription. The Configuration of the Logging service section guide shows how to get this value. |
Info |
---|
This parameter is optional, but you must select one between |
Info |
---|
This parameter should be removed if it is not used. |
subscription_id
str
Mandatory
Minimum length: 1
The ID of the Pub/sub subscription. The Configuration of the Logging service section guide shows how to get this value.
custom_tag
str
Optional
|
|
| A Devo tag. See for more information see Devo tags |
Recommended custom_tag for the | There are three possible scenarios:
|
If a custom_tag is not defined, all events that are not in |
This parameter should be removed if it is not used. | ||||||
|
|
| It can have one of these two values:
| The default value of this parameter is
|
This parameter should be removed if it is not used. | ||||
|
|
| Minimum length: 1 | Period in seconds used between each data pulling, this value will overwrite the default value (60 seconds) |
This parameter should be removed if it is not used. | ||||||
|
|
|
| Use this param to enable/disable the undelivered messages logging feature, that will show how many undelivered messages remains in the GCP pubsub queue every N data pulls.
|
This parameter should be removed if it is not used. | ||||||
|
|
| Minimum value: 1 | This param, defines (when enabled) every how many seconds will be trigered the undelivered messages log line.
|
This parameter should be removed if it is not used. |
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Collector Docker image
SHA-256 hash
7a75133fed4e89de782f8b7b04599fbd6f5ef8c2663789a52fec3493cd9f7d5f
Rw tab | ||
---|---|---|
|
This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.
Structure
The following directory structure should be created for being used when running the collector:
Code Block |
---|
<any_directory> └── devo-collectors/ └── <product_name>/ ├── certs/ │ ├── chain.crt │ ├── <your_domain>.key │ └── <your_domain>.crt ├── state/ -└── ./certs:/devo-collector/certsconfig/ - ./config:/devo-collector/config └── - ./credentials:/devo-collector/credentials - ./state:/devo-collector/state environment: - CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/
directory:
Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace |
Rw tab | ||
---|---|---|
|
We use a piece of software called Collector Server to host and manage all our available collectors.
To enable the collector for a customer:
In the Collector Server GUI, access the domain in which you want this instance to be created
Click Add Collector and find the one you wish to add.
In the Version field, select the latest value.
In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).
In the sending method select Direct Send. Direct Send configuration is optional for collectors that create
Table
events, but mandatory for those that createLookups
.In the Parameters section, establish the Collector Parameters as follows below:
Editing the JSON configuration
Code Block |
---|
{
"global_overrides": {
"debug": false
},
"inputs": {
"gcp": {
"id": "<short_unique_id>",
"enabled": <input_status>,
"requests_per_second": <requests_per_second>,
"credentials": {
"source_id": "<source_id_value>",
"project_id": "<project_id_value>",
"filename": "<filename_value>",
"file_content_base64": "<file_content_base64>"
},
"services": {
"logging": {
"subscription_name": "<subscription_id>",
"tag": "<custom_tag>",
"auto_categorization_method": "<auto_categorization_method>",
"request_period_in_seconds": <request_period_in_seconds>,
"enable_pubsub_undelivered_messages_logging": <enable_pubsub_undelivered_messages_logging>,
"pubsub_undelivered_messages_request_interval_in_seconds": <pubsub_undelivered_messages_request_interval_in_seconds>
},
"scc-findings": {
"subscription_name": "<subscription_id>",
"tag": "<custom_tag>",
"request_period_in_seconds": <request_period_in_seconds>
}
}
}
}
} |
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the |
Please replace the placeholders with real world values following the description table below:
Parameter
Data Type
Type
Value Range / Format
Details
short_unique_id
int
Mandatory
Minimum length: 1
Maximum length: 5
Use this param to give an unique id to this input service.
Note |
---|
This parameter is used to build the persistence address, do not use the same value for multiple collectors. It could cause a collision. |
input_status
bool
Mandatory
false
/ true
Use this param to enable or disable the given input logic when running the collector. If the value is true
, the input will be run. If the value is false
, it will be ignored.
requests_per_second
int
Optional
Minimum value: 1
100000
requests/sec.config.yaml |
Note |
---|
Replace |
Devo credentials
In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <product_name>/certs/
. Learn more about security credentials in Devo here.
Note |
---|
Replace |
Editing the config.yaml file
Code Block |
---|
globals:
debug: false
id: <collector_id>
name: <collector_name>
persistence:
type: filesystem
config:
directory_name: state
outputs:
devo_1:
type: devo_platform
config:
address: <devo_address>
port: 443
type: SSL
chain: <chain_filename>
cert: <cert_filename>
key: <key_filename>
inputs:
gcp:
id: <short_unique_id>
enabled: <input_status>
requests_per_second: <requests_per_second>
credentials:
source_id: <source_id_value>
project_id: <project_id_value>
filename: <filename_value>
file_content_base64: <file_content_base64_value>
services:
logging:
subscription_name: <subscription_id>
tag: <custom_tag>
auto_categorization_method: <auto_categorization_method>
request_period_in_seconds: <request_period_in_seconds>
enable_pubsub_undelivered_messages_logging: <enable_pubsub_undelivered_messages_logging>
pubsub_undelivered_messages_request_interval_in_seconds: <pubsub_undelivered_messages_request_interval_in_seconds>
scc-findings:
subscription_name: <subscription_id>
tag: <custom_tag>
request_period_in_seconds: <request_period_in_seconds> |
Info |
---|
All defined service entities will be executed by the collector. If you do not want to run any of them, just remove the entity from the |
Replace the placeholders with your required values following the description table below:
Maximum length: 5
Parameter | Data Type | Type | Value Range / Format | Details | ||||||
|
|
| Minimum length: 1 | Use this param to give an unique id to this input service.
| ||||||
|
|
|
| Use this param to enable or disable the given input logic when running the collector. If the value is | ||||||
|
|
| Minimum value: 1 | Customize the maximum number of API requests per second. If not used, the default setting will be used:
| ||||||
|
|
| Minimum length: 1 | Use this param to assign a custom name for identifying the environment of the infrastructure.
| ||||||
|
|
| Minimum length: 1 | The name of the GCP project. The Configuration of the Logging service section guide shows how to get this value.
| ||||||
|
|
| Minimum length: 1 | The file name with the account credentials. The Configuration of the Logging service section guide shows how to get this value.
| ||||||
|
|
| Minimum length: 1 | The Service Account credentials in base64. The Configuration of the Logging service section guide shows how to get this value.
| ||||||
|
|
| Minimum length: 1 | The ID of the Pub/sub subscription. The Configuration of the Logging service section guide shows how to get this value. | ||||||
|
|
| A Devo tag. See for more information see Devo tags
| There are three possible scenarios:
| ||||||
|
|
| It can have one of these two values:
| The default value of this parameter is
| ||||||
|
|
| Minimum length: 1 | Period in seconds used between each data pulling, this value will overwrite the default value (60 seconds)
|
source_id_value
int
Mandatory
|
|
|
| Use this param to |
Info |
---|
This |
project_id_value
str
Mandatory
Minimum length: 1
The name of the GCP project. The Configuration of the Logging service section guide shows how to get this value.
Info |
---|
This |
filename_value
str
Optional
Minimum length: 1
The file name with the account credentials. The Configuration of the Logging service section guide shows how to get this value.
Info |
---|
This parameter is optional, but you must select one between |
enable/disable the undelivered messages logging feature, that will show how many undelivered messages remains in the GCP pubsub queue every N data pulls.
| ||||||||
|
|
| Minimum value: 1 | This param, defines (when enabled) every how many seconds will be trigered the undelivered messages log line.
|
file_content_base64_value
str
Optional
Minimum length: 1
The Service Account credentials in base64. The Configuration of the Logging service section guide shows how to get this value.
Info |
---|
This parameter is optional, but you must select one between |
This parameter should be removed if it is not used.
subscription_id
str
Mandatory
Minimum length: 1
The ID of the Pub/sub subscription. The Configuration of the Logging service section guide shows how to get this value.
custom_tag
str
Optional
A Devo tag. See for more information see Devo tags
Recommended custom_tag for the scc-findings
service:cloud.gcp.scc.findings
There are three possible scenarios:
Send all events to custom tag: Add the desired value to the custom_tag
Send the events to the default tag in the Logging service: Do not use the custom_tag
Send SCC findings events to their corresponding tag: add the
cloud.gcp.scc.findings
value to the custom tag inscc-findings
service.
If a custom_tag is not defined, all events that are not in JSON
format will be ingested to the tag cloud.gcp.unknown.none
This parameter should be removed if it is not used.
auto_categorization_method
str
Optional
It can have one of these two values:
logname_field
resource_type_field
The default value of this parameter islogname_field.
It is used only to set the auto-categorization method to resource_type_field
.
Info |
---|
Visit the Devo categorization and destination section for details. |
This parameter should be removed if it is not used.
request_period_in_seconds
int
Optional
Minimum length: 1
Period in seconds used between each data pulling, this value will overwrite the default value (60 seconds)
This parameter should be removed if it is not used.
enable_pubsub_undelivered_messages_logging
bool
Optional
false
/ true
Use this param to enable/disable the undelivered messages logging feature, that will show how many undelivered messages remains in the GCP pubsub queue every N data pulls.
Default value: false
Note |
---|
Bear in mind that this feature uses the monitoring API, that might have a cost if we exceed a certain number of requests. |
This parameter should be removed if it is not used.
pubsub_undelivered_messages_request_interval_in_seconds
int
Optional
Minimum value: 1
This param, defines (when enabled) every how many seconds will be trigered the undelivered messages log line.
Default value: 300
Note |
---|
Bear in mind that this feature uses the monitoring API, that might have a cost if we exceed a certain number of requests. |
Download the Docker image
The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:
Collector Docker image | SHA-256 hash |
---|---|
|
Use the following command to add the Docker image to the system:
Code Block |
---|
gunzip -c <image_file>-<version>.tgz | docker load |
Note |
---|
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace |
The Docker image can be deployed on the following services:
Docker
Execute the following command on the root directory <any_directory>/devo-collectors/<product_name>/
Code Block |
---|
docker run
--name collector-<product_name>
--volume $PWD/certs:/devo-collector/certs
--volume $PWD/config:/devo-collector/config
--volume $PWD/state:/devo-collector/state
--env CONFIG_FILE=config.yaml
--rm
--interactive
--tty
<image_name>:<version> |
Note |
---|
Replace |
Docker Compose
The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/<product_name>/
directory.
Code Block |
---|
version: '3'
services:
collector-<product_name>:
image: <image_name>:${IMAGE_VERSION:-latest}
container_name: collector-<product_name>
volumes:
- ./certs:/devo-collector/certs
- ./config:/devo-collector/config
- ./credentials:/devo-collector/credentials
- ./state:/devo-collector/state
environment:
- CONFIG_FILE=${CONFIG_FILE:-config.yaml} |
To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/<product_name>/
directory:
Code Block |
---|
IMAGE_VERSION=<version> docker-compose up -d |
Note |
---|
Replace |
Collector services detail
...
Expand | ||||
---|---|---|---|---|
| ||||
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
|
Expand | ||
---|---|---|
| ||
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode. |
Change log
...
Release
...
Released on
...
Release type
...
Details
...
|
Expand | ||
---|---|---|
| ||
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode. |
Change log
Release | Released on | Release type | Details | Recommendations | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| Improvements
|
| |||||||||||||
|
| Improvements
|
| |||||||||||||
|
| Improvements
New Featuresfeatures
|
| |||||||||||||
|
| Improvements:
|
| |||||||||||||
|
| Improvements:
Bug fixing:
|
| |||||||||||||
|
| - | - | - | ||||||||||||
|
|
| Improvements:
Bug Fixing:fixing
|
| ||||||||||||
|
| Improvements:
|
| |||||||||||||
|
| Improvements:
|
| |||||||||||||
|
| Improvements:
Vulnerabilities mitigated:
|
| |||||||||||||
|
| Improvements:
|
| |||||||||||||
|
| Improvements:
|
|