Google Cloud Platform collector

Service description

This collector will retrieve the information stored in Google Cloud Platform (GCP), such as audit logs, networking, load balance, and more.

Data source description

The Google Cloud Platform (GCP) centralizes all the monitoring information from all services in the cloud catalog inside a service called Stackdriver.

There is certain information that is enabled by default and is free of charge. However, generating some other information will imply some costs and that's why it must be enabled manually. In both cases, the generated information (we will call them messages) will arrive at the Stackdriver service.

There exist many more GCP services than the ones shown in the diagram, these are only a small sample.

The Stackdriver service has different ways of exporting the information stored inside (structured in messages). In this case, it uses another GCP service called PubSub, which contains a "Topic" object that will receive a filtered set of messages from the Stackdriver service. Then, the GCP collector will retrieve all those messages from the "Topic" object using a subscription (concretely in "pull" mode):

The GCP collector that we provide processes the stored information and sends it to the Devo platform. Data will be sent to a central table within your Devo domain named cloud.gcp.

Service name definitions

Listed in the table below are the service names, details, and how the Devo platform treats the data.

Services	Details	Devo tables
Audit Resource	Get information on Cloud Audit Logs that maintain three audit logs for each Google Cloud project, folder, and organization: Admin Activity, Data Access, and System Event. Google Cloud services write audit log entries to these logs to help you answer the questions of "who did what, where, and when?" within your Google Cloud resources.	`cloud.gcp.logging.audited_resource`
Virtual Machines	Get information about the Virtual Machine Instance, CPU, Memory, and more.	`cloud.gcp.logging.gce_instance`
Redis	Get information on memory store for Redis instance's performance, and allows you to check in on your instances to make sure they're behaving as desired.	`cloud.gcp.logging.redis`
Networking	Get information on memory store usage.	`cloud.gcp.logging.gce_subnetwork`
Load Balance	Get information about the load balancer that distributes user traffic across multiple instances of your applications.	`cloud.gcp.logging.load_balancer`
Kubernetes Engine	Get information about Google Kubernetes Engine (GKE), which provides a managed environment for deploying, managing, and scaling your containerized applications using Google infrastructure.	`cloud.gcp.logging.k8s`
PubSub	Get information about the asynchronous messaging service.	`cloud.gcp.logging.pubsub`

Setup

The collector will be executed inside of the Collector Server as well as a Standalone mode (docker container). Some configurations will be necessary for the proper functioning of the collector.

Source setup types

In order to have the GCP collector running in a proper way, the data source (in this case GCP) must have several service structures configured. Specifically, it is required to set subscription objects to pull mode.

The required GCP setup can be created either automatically or manually. Depending on a configuration file property called autoconfig, one mode or another will be used and the behavior will be the following:

Automatic

The autoconfig property must have the enabled option set to true. Also, the internal service definition should have a configuration class value set (this cannot be set by the customer). The autoconfiguration functionality will create different "sinks" using the predefined service definition or using some custom definitions from the configuration file.

Manual

The autoconfig property must have the enabled option set to false and any configuration class defined in the internal service definition will be ignored. Depending if the property subscription_name is set, two different behaviors will be applied:

If subscription_name is defined, the value will be used for connecting to the GCP subscription. If the subscription doesn't exist, it will cause an eternal loop checking every minute if that subscription exists (the console logs will show a warning).
If subscription_name is not defined, a subscription with the same name as the one auto-calculated by the autoconfiguration functionality will be expected.

Service types

The GCP collector must have at least one service entry in the services section. These services can be of two types:

Predefined services

These services will not require to set some properties in the configuration file since they are already internally defined. This is the list of predefined services:

virtual_machines → Resources from "Compute Engine" GCP service.
networking → Resources from "VPC network" GCP service.
load_balancing → Resources from "Network services > Load balancing" GCP service.
pubsub → Resources from "PubSub" service.
redis → Resources from "Redis" service.

Custom services

These services will require some extra properties in the configuration file:

sink_filter_resource → This is the value of the filter that will be applied to the GCP Sink object when the all value is set in the regions section during the automatic configuration phase (if enabled).
sink_filter_resource_region → This is the value of the filter that will be applied to the GCP Sink object when any value distinct from all is set in the regions section during the autoconfiguration phase (if enabled). The filter must contain a placeholder with value {region} as part of its value.

The syntax to be used in sink_filter_resource_* properties is detailed here.

Credentials

Follow the next steps to create the Service Account that will be used to collect the alerts and enable the necessary API and scopes to use it.

Go to your Google GCP console project and open the left menu. Click IAM & Admin → Service Accounts.
Click on + Create Service Account to create the credentials.
Follow the 3 steps to create the credentials. Enter a name and click on Create. Steps 2 and 3 are optional, so you may just click Continue to go to step 3 and select Done.
Now you have to add the Keys to the service account that was previously created and download it in JSON format. After clicking the Done button, you’ll be redirected to the services accounts of your project. Search the service account that you created and click it.
In the Service account details area, click ADD KEY → Create new key, select the JSON format, and click CREATE. Download the credentials file and move it to the <any_directory>/devo-collectors/gcp/credentials/ directory.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).