{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"minLevel":{"value":"2"},"maxLevel":{"value":"2"},"type":{"value":"flat"}},"macroMetadata":{"macroId":{"value":"b07f9d852462fbbf42eed4e0b419c636"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"0530ce08-5d6d-4d82-ac96-13ea838a5aee"}},{"type":"heading","attrs":{"level":2},"content":[{"text":"Service description","type":"text"}]},{"type":"paragraph","content":[{"text":"This collector will retrieve the information stored in ","type":"text"},{"text":"Google Cloud Platform","type":"text","marks":[{"type":"link","attrs":{"href":"https://cloud.google.com/gcp/?utm_source=google&utm_medium=cpc&utm_campaign=emea-es-all-en-dr-bkws-all-all-trial-e-gcp-1008073&utm_content=text-ad-none-any-DEV_c-CRE_167377618211-ADGP_Hybrid%20%7C%20AW%20SEM%20%7C%20BKWS%20~%20EXA_M%3A1_ES_EN_General_Cloud_gcp%20google-KWID_43700053286673525-kwd-62537304604-userloc_1005490&utm_term=KW_gcp%20google-NET_g-PLAC_&ds_rl=1242853&ds_rl=1245734&gclid=EAIaIQobChMIqsyvyfX46AIVguFRCh3qiwySEAAYASAAEgLHY_D_BwE"}}]},{"text":" (GCP), such as audit logs, networking, load balance, and more.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Data source description","type":"text"}]},{"type":"paragraph","content":[{"text":"The Google Cloud Platform (GCP) centralizes all the monitoring information from all services in the cloud catalog inside a service called Stackdriver.","type":"text"}]},{"type":"paragraph","content":[{"text":"There is certain information that is enabled by default and is free of charge. However, generating some other information will imply some costs and that's why it must be enabled manually. In both cases, the generated information (we will call them ","type":"text"},{"text":"messages","type":"text","marks":[{"type":"em"}]},{"text":") will arrive at the Stackdriver service.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":65.0},"content":[{"type":"media","attrs":{"width":1372,"id":"9778e620-0959-43ec-8983-908103d6e82e","collection":"contentId-94638516","type":"file","height":1052}}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"There exist many more GCP services than the ones shown in the diagram, these are only a small sample.","type":"text"}]}]},{"type":"paragraph","content":[{"text":"The Stackdriver service has different ways of exporting the information stored inside (structured in messages). In this case, it uses another GCP service called PubSub, which contains a \"Topic\" object that will receive a filtered set of messages from the Stackdriver service. Then, the GCP collector will retrieve all those messages from the \"Topic\" object using a subscription (concretely in \"pull\" mode):","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":30.0},"content":[{"type":"media","attrs":{"width":804,"id":"5defe31c-f899-44cf-8de1-940bdec72062","collection":"contentId-94638516","type":"file","height":452}}]},{"type":"paragraph","content":[{"text":"The GCP collector that we provide processes the stored information and sends it to the Devo platform. Data will be sent to a central table within your Devo domain named ","type":"text"},{"text":"cloud.gcp","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":30.0},"content":[{"type":"media","attrs":{"width":924,"id":"54812637-5f43-41cd-86a0-54abb6885bde","collection":"contentId-94638516","type":"file","height":452}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Service name definitions","type":"text"}]},{"type":"paragraph","content":[{"text":"Listed in the table below are the service names, details, and how the Devo platform treats the data.","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"25d6204e-e12c-41bd-8722-f26331cc2cee"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Services","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Details","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Devo tables","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Audit Resource","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Get information on Cloud ","type":"text"},{"text":"Audit Logs","type":"text","marks":[{"type":"strong"}]},{"text":" that maintain three ","type":"text"},{"text":"audit logs","type":"text","marks":[{"type":"strong"}]},{"text":" for each Google Cloud project, folder, and organization: Admin Activity, Data Access, and System Event. Google Cloud services write ","type":"text"},{"text":"audit log","type":"text","marks":[{"type":"strong"}]},{"text":" entries to these ","type":"text"},{"text":"logs","type":"text","marks":[{"type":"strong"}]},{"text":" to help you answer the questions of \"who did what, where, and when?\" within your Google Cloud ","type":"text"},{"text":"resources","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cloud.gcp.logging.audited_resource","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Virtual Machines","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Get information about the Virtual Machine Instance, CPU, Memory, and more.","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cloud.gcp.logging.gce_instance","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Redis","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Get information on memory store for Redis instance's performance, and allows you to check in on your instances to make sure they're behaving as desired.","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cloud.gcp.logging.redis","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Networking","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Get information on memory store usage.","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cloud.gcp.logging.gce_subnetwork","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Load Balance","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Get information about the load balancer that distributes user traffic across multiple instances of your applications.","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cloud.gcp.logging.load_balancer","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Kubernetes Engine","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Get information about Google Kubernetes Engine (GKE), which provides a managed environment for deploying, managing, and scaling your containerized applications using Google infrastructure.","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cloud.gcp.logging.k8s","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"PubSub","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"Get information about the asynchronous messaging service.","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[226.67]},"content":[{"type":"paragraph","content":[{"text":"cloud.gcp.logging.pubsub","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Setup","type":"text"}]},{"type":"paragraph","content":[{"text":"The collector will be executed inside of the Collector Server as well as a Standalone mode (docker container). Some configurations will be necessary for the proper functioning of the collector.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Source setup types","type":"text"}]},{"type":"paragraph","content":[{"text":"In order to have the GCP collector running in a proper way, the data source (in this case GCP) must have several service structures configured. Specifically, it is required to set ","type":"text"},{"text":"subscription","type":"text","marks":[{"type":"strong"}]},{"text":" objects to pull mode.","type":"text"}]},{"type":"paragraph","content":[{"text":"The required GCP setup can be created either automatically or manually. Depending on a configuration file property called ","type":"text"},{"text":"autoconfig","type":"text","marks":[{"type":"strong"}]},{"text":", one mode or another will be used and the behavior will be the following:","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"24aafc5a-6818-4bc9-9d3b-6df10726d4bd"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Automatic","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"autoconfig","type":"text","marks":[{"type":"strong"}]},{"text":" property must have the ","type":"text"},{"text":"enabled","type":"text","marks":[{"type":"strong"}]},{"text":" option set to ","type":"text"},{"text":"true","type":"text","marks":[{"type":"em"}]},{"text":". Also, the internal service definition should have a configuration class value set (this cannot be set by the customer). The autoconfiguration functionality will create different \"sinks\" using the predefined service definition or using some custom definitions from the configuration file.","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Manual","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"autoconfig ","type":"text","marks":[{"type":"strong"}]},{"text":"property must have the ","type":"text"},{"text":"enabled","type":"text","marks":[{"type":"strong"}]},{"text":" option set to ","type":"text"},{"text":"false","type":"text","marks":[{"type":"em"}]},{"text":" and any configuration class defined in the internal service definition will be ignored. Depending if the property ","type":"text"},{"text":"subscription_name","type":"text","marks":[{"type":"strong"}]},{"text":" is set, two different behaviors will be applied:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"subscription_name","type":"text","marks":[{"type":"strong"}]},{"text":" is defined, the value will be used for connecting to the GCP subscription. If the subscription doesn't exist, it will cause an eternal loop checking every minute if that subscription exists (the console logs will show a warning).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"If ","type":"text"},{"text":"subscription_name","type":"text","marks":[{"type":"strong"}]},{"text":" is not defined, a subscription with the same name as the one auto-calculated by the autoconfiguration functionality will be expected.","type":"text"}]}]}]}]}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Service types","type":"text"}]},{"type":"paragraph","content":[{"text":"The GCP collector must have at least one service entry in the ","type":"text"},{"text":"services","type":"text","marks":[{"type":"strong"}]},{"text":" section. These services can be of two types:","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"5e054322-f935-4011-96dc-bb328d45c727"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Predefined services","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"These services will not require to set some properties in the configuration file since they are already internally defined. This is the list of predefined services:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"virtual_machines → Resources from \"Compute Engine\" GCP service.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"networking → Resources from \"VPC network\" GCP service.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"load_balancing → Resources from \"Network services > Load balancing\" GCP service.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"pubsub → Resources from \"PubSub\" service.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"redis → Resources from \"Redis\" service.","type":"text"}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Custom services","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"These services will require some extra properties in the configuration file:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"sink_filter_resource → This is the value of the filter that will be applied to the ","type":"text"},{"text":"GCP Sink","type":"text","marks":[{"type":"strong"}]},{"text":" object when the ","type":"text"},{"text":"all","type":"text","marks":[{"type":"strong"}]},{"text":" value is set in the ","type":"text"},{"text":"regions","type":"text","marks":[{"type":"strong"}]},{"text":" section during the automatic configuration phase (if enabled).","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"sink_filter_resource_region → This is the value of the filter that will be applied to the ","type":"text"},{"text":"GCP Sink","type":"text","marks":[{"type":"strong"}]},{"text":" object when any value distinct from ","type":"text"},{"text":"all","type":"text","marks":[{"type":"strong"}]},{"text":" is set in the ","type":"text"},{"text":"regions","type":"text","marks":[{"type":"strong"}]},{"text":" section during the autoconfiguration phase (if enabled). The filter must contain a ","type":"text"},{"text":"placeholder","type":"text","marks":[{"type":"strong"}]},{"text":" with value ","type":"text"},{"text":"{region}","type":"text","marks":[{"type":"em"}]},{"text":" as part of its value.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The syntax to be used in sink_filter_resource_* properties is detailed ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://cloud.google.com/logging/docs/view/advanced-queries"}}]},{"text":".","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Credentials","type":"text"}]},{"type":"paragraph","content":[{"text":"Follow the next steps to create the Service Account that will be used to collect the alerts and enable the necessary API and scopes to use it.","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Go to your ","type":"text"},{"text":"Google GCP console","type":"text","marks":[{"type":"link","attrs":{"href":"https://console.cloud.google.com/home/dashboard"}}]},{"text":" project and open the left menu. Click ","type":"text"},{"text":"IAM & Admin","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"→","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"Service Accounts","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"},{"type":"hardBreak"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":55.0},"content":[{"type":"media","attrs":{"width":3810,"id":"d8b300d6-564a-4b61-b133-0cb167d980c7","collection":"contentId-94638516","type":"file","height":2064}}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Click on ","type":"text"},{"text":"+","type":"text","marks":[{"type":"strong"}]},{"text":" ","type":"text"},{"text":"Create Service Account ","type":"text","marks":[{"type":"strong"}]},{"text":"to create the credentials.","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":55.0},"content":[{"type":"media","attrs":{"width":3300,"id":"b57ec9b7-8179-4dc3-8b32-34f9b8d6b060","collection":"contentId-94638516","type":"file","height":1270}}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Follow the 3 steps to create the credentials. Enter a ","type":"text"},{"text":"name ","type":"text","marks":[{"type":"strong"}]},{"text":"and click on ","type":"text"},{"text":"Create","type":"text","marks":[{"type":"strong"}]},{"text":". Steps 2 and 3 are optional, so you may just click ","type":"text"},{"text":"Continue","type":"text","marks":[{"type":"strong"}]},{"text":" to go to step 3 and select ","type":"text"},{"text":"Done","type":"text","marks":[{"type":"strong"}]},{"text":".","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":55.0},"content":[{"type":"media","attrs":{"width":3010,"id":"5afe6c0b-f619-440b-a516-65435dea5172","collection":"contentId-94638516","type":"file","height":1252}}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Now you have to add the ","type":"text"},{"text":"Keys ","type":"text","marks":[{"type":"strong"}]},{"text":"to the service account that was previously created","type":"text"},{"text":" ","type":"text","marks":[{"type":"strong"}]},{"text":"and download it in JSON format. After clicking the ","type":"text"},{"text":"Done ","type":"text","marks":[{"type":"strong"}]},{"text":"button, you’ll be redirected to the services accounts of your project. Search the service account that you created and click it.","type":"text"},{"type":"hardBreak"}]},{"type":"mediaSingle","attrs":{"layout":"center","width":45.0},"content":[{"type":"media","attrs":{"width":3278,"id":"880b61b2-4709-4e45-8cef-9cc432df7856","collection":"contentId-94638516","type":"file","height":1284}}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"In the ","type":"text"},{"text":"Service account details ","type":"text","marks":[{"type":"strong"}]},{"text":"area, click ","type":"text"},{"text":"ADD KEY → Create new key","type":"text","marks":[{"type":"strong"}]},{"text":", select the ","type":"text"},{"text":"JSON","type":"text","marks":[{"type":"strong"}]},{"text":" format, and click ","type":"text"},{"text":"CREATE","type":"text","marks":[{"type":"strong"}]},{"text":". Download the credentials file and move it to the ","type":"text"},{"text":"/devo-collectors/gcp/credentials/","type":"text","marks":[{"type":"code"}]},{"text":" directory.","type":"text"}]}]}]},{"type":"layoutSection","content":[{"type":"layoutColumn","attrs":{"width":50.0},"content":[{"type":"mediaSingle","attrs":{"layout":"center","width":100.0},"content":[{"type":"media","attrs":{"width":3066,"id":"b09afdbe-b5d0-44d4-8e3e-695da4d42626","collection":"contentId-94638516","type":"file","height":1656}}]}]},{"type":"layoutColumn","attrs":{"width":50.0},"content":[{"type":"mediaSingle","attrs":{"layout":"center","width":100.0},"content":[{"type":"media","attrs":{"width":2870,"id":"f4f74eb6-ad66-4d5b-a552-6109d3521e72","collection":"contentId-94638516","type":"file","height":1616}}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Run the collector","type":"text"}]},{"type":"paragraph","content":[{"text":"Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (","type":"text"},{"text":"Cloud collector","type":"text","marks":[{"type":"underline"}]},{"text":"), or deploy and host the collector in your own machine using a Docker image (","type":"text"},{"text":"On-premise collector","type":"text","marks":[{"type":"underline"}]},{"text":").","type":"text"}]},{"type":"bodiedExtension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"rw-ui-tabs-macro","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"07ff7a9e-7d96-4fa6-a129-4fc89fd35a4c"},"schemaVersion":{"value":"1"},"placeholder":[{"type":"icon","data":{"url":"https://d1bgdtsfe1sjdg.cloudfront.net/refinedtheme-for-cloud/macro-icons/tabs.png"}}],"title":"UI Tabs"}},"localId":"70cb00af-2a43-4cf0-87d5-5764763941b3"},"content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"rw-tab","parameters":{"macroParams":{"title":{"value":"Cloud collector"}},"macroMetadata":{"macroId":{"value":"f4dedef800ae72487d63cc62df3896bb"},"schemaVersion":{"value":"1"},"placeholder":[{"type":"icon","data":{"url":"https://d1bgdtsfe1sjdg.cloudfront.net/refinedtheme-for-cloud/macro-icons/tab.png"}}],"title":"UI Tab"}},"localId":"d674bda7-6658-4b96-9d8a-495fed5c9ad1"}},{"type":"paragraph","content":[{"text":"We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, ","type":"text"},{"text":"get in touch with us","type":"text","marks":[{"type":"link","attrs":{"href":"mailto:support@devo.com"}}]},{"text":" and we will guide you through the configuration.","type":"text"}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"rw-tab","parameters":{"macroParams":{"title":{"value":"On-premise collector"}},"macroMetadata":{"macroId":{"value":"cbed9ece07a9d571ddf7c19ef06a7bc5"},"schemaVersion":{"value":"1"},"placeholder":[{"type":"icon","data":{"url":"https://d1bgdtsfe1sjdg.cloudfront.net/refinedtheme-for-cloud/macro-icons/tab.png"}}],"title":"UI Tab"}},"localId":"5d0a6076-57ff-4211-8cfe-246266cc6e30"}},{"type":"paragraph","content":[{"text":"This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Structure","type":"text"}]},{"type":"paragraph","content":[{"text":"The following directory structure will be required as part of the setup procedure (it can be created under any directory):","type":"text"}]},{"type":"codeBlock","content":[{"text":"\n└── devo-collectors/\n └── gcp/\n ├── certs/\n │ ├── chain.crt\n │ ├── .key\n │ └── .crt\n ├── credentials/\n │ └── credentials-gcp.json\n └── config/\n └── config-gcp.yaml","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Devo credentials","type":"text"}]},{"type":"paragraph","content":[{"text":"In Devo, go to ","type":"text"},{"text":"Administration → Credentials → X.509 Certificates","type":"text","marks":[{"type":"strong"}]},{"text":", download the ","type":"text"},{"text":"Certificate","type":"text","marks":[{"type":"strong"}]},{"text":", ","type":"text"},{"text":"Private key","type":"text","marks":[{"type":"strong"}]},{"text":" and ","type":"text"},{"text":"Chain CA","type":"text","marks":[{"type":"strong"}]},{"text":" and save them in ","type":"text"},{"text":"/devo-collectors/gcp/certs/","type":"text","marks":[{"type":"code"}]},{"text":". Learn more about security credentials in Devo ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://devodocs.atlassian.net/wiki/pages/createpage.action?spaceKey=NDT5&title=Security%20credentials"}}]},{"text":". ","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"center"},"content":[{"type":"media","attrs":{"width":1840,"id":"4f63f799-b9db-463f-9b14-e30cf438213a","collection":"contentId-94638516","type":"file","height":414}}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Editing the config-gcp.yaml file","type":"text"}]},{"type":"paragraph","content":[{"text":"In the ","type":"text"},{"text":"config-gcp.yaml","type":"text","marks":[{"type":"strong"}]},{"text":" file, replace the ","type":"text"},{"text":", , , ","type":"text","marks":[{"type":"code"}]},{"text":"and ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":" values and enter the ones that you got in the previous steps. In the ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":" placeholder, enter the value that you choose.","type":"text"}]},{"type":"paragraph","content":[{"text":"The GCP collector has many different configurations. See some usage config examples and custom services below.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Predefined","type":"text","marks":[{"type":"strong"}]},{"text":" services and ","type":"text"},{"text":"autoconfig","type":"text","marks":[{"type":"strong"}]},{"text":" functionality ","type":"text"},{"text":"enabled","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"globals:\n debug: false\n id: not_used\n name: gcp\n persistence:\n type: filesystem # File system persistence ON\n config:\n directory_name: state # Directory where the persistence will be saved in case of using filesystem\noutputs:\n devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)\n type: devo_platform\n config:\n address: collector-eu.devo.io # URL to use for sending messages\n port: 443 # SSL for using certificates or TCP for sending without encryption\n type: SSL\n chain: chain.crt # \\\n cert: .crt # Credential files downloaded from a Devo domain\n key: .key # /\ninputs:\n gcp:\n id: # The value of this field will be used internally for having independent persistence areas\n enabled: true # If \"false\" value this \"input\" will not be used by collector\n requests_per_second: 5 # Setup how many request API por second\n use_time_slots_for_retrievals: false\n autoconfig:\n enabled: true # Determines if the \"autoconfiguration\" functionality will be executed\n refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended.\n credentials:\n source_id: # This value will be used for adding to message \"tag\" as fourth level\n project_id: # This value will be used for adding to message \"tag\" as fifth level\n filename: # Credentials file generated from the \"GCP administration console\"\n file_content_base64: # Credentials file content generated from the \"GCP administration console\" in base64 format\n services: # Services available for this collector are virtual machines, networking, load balance, redis and PubSub.\n virtual_machines:\n request_period_in_seconds: 15 # Setting up time interval between API requests. 15sec is recommended.\n regions: # Regions for being used as filter of source data, use \"all\" value for no filter by region\n - all\n networking:\n request_period_in_seconds: 15\n regions:\n - all\n load_balancing:\n request_period_in_seconds: 15\n regions:\n - all\n pubsub:\n request_period_in_seconds: 15\n regions:\n - all\n redis:\n request_period_in_seconds: 15\n regions:\n - all\n custom_service_1:\n tag_subtype: \"all1\"\n subscription_name: \"custom_service_1-subscription\"\n request_period_in_seconds: 30\n custom_service_2:\n tage: \"my.app.gcp.test1\"\n subscription_name: \"custom_service_2-subscription\"\n request_period_in_seconds: 30","type":"text"}]},{"type":"paragraph","content":[{"text":"The custom configuration section can be set with many different values. Find below several examples of the custom configuration section and their impact on the different services involved.","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Predefined","type":"text","marks":[{"type":"strong"}]},{"text":" services and ","type":"text"},{"text":"autoconfig","type":"text","marks":[{"type":"strong"}]},{"text":" functionality ","type":"text"},{"text":"disabled","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"globals:\n debug: false\n id: not_used\n name: gcp\n persistence:\n type: filesystem # File system persistence ON\n config:\n directory_name: state # Directory where the persistence will be saved in case of using filesystem\noutputs:\n devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)\n type: devo_platform\n config:\n address: collector-eu.devo.io # URL to use for sending messages\n port: 443 # SSL for using certificates or TCP for sending without encryption\n type: SSL\n chain: chain.crt # \\\n cert: .crt # Credential files downloaded from a Devo domain\n key: .key # /\ninputs:\n gcp:\n id: # The value of this field will be used internally for having independent persistence areas\n enabled: true # If \"false\" value this \"input\" will not be used by collector\n requests_per_second: 5 # Setup how many request API por second\n use_time_slots_for_retrievals: false\n autoconfig:\n enabled: false # Determines if the \"autoconfiguration\" functionality will be executed\n refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended.\n credentials:\n source_id: # This value will be used for adding to message \"tag\" as fourth level\n project_id: # This value will be used for adding to message \"tag\" as fifth level\n filename: # Credentials file generated from the \"GCP administration console\"\n file_content_base64: # Credentials file content generated from the \"GCP administration console\" in base64 format\n services: # Services available for this collector are virtual machines, networking, load balance, redis and PubSub.\n virtual_machines:\n request_period_in_seconds: 15 # Setting up time interval between API requests. 15sec is recommended.\n regions: \n - europe-west2 # Regions for being used as filters of source data\n - europe-west3\n networking:\n request_period_in_seconds: 15\n regions:\n - europe-west3\n load_balancing:\n request_period_in_seconds: 15\n regions:\n - all\n pubsub:\n request_period_in_seconds: 15\n regions:\n - all\n redis:\n request_period_in_seconds: 15\n regions:\n - all\n custom_service_1:\n tag_subtype: \"all1\"\n subscription_name: \"custom_service_1-subscription\"\n request_period_in_seconds: 30\n custom_service_2:\n tag: \"my.app.gcp.test1\"\n subscription_name: \"custom_service_2-subscription\"\n request_period_in_seconds: 30","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Custom ","type":"text","marks":[{"type":"strong"}]},{"text":"services, ","type":"text"},{"text":"autoconfig","type":"text","marks":[{"type":"strong"}]},{"text":" functionality ","type":"text"},{"text":"enabled ","type":"text","marks":[{"type":"strong"}]},{"text":"and using ","type":"text"},{"text":"regions","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"globals:\n debug: false\n id: not_used\n name: gcp\n persistence:\n type: filesystem # File system persistence ON\n config:\n directory_name: state # Directory where the persistence will be saved in case of using filesystem\noutputs:\n devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)\n type: devo_platform\n config:\n address: collector-eu.devo.io # URL to use for sending messages\n port: 443 # SSL for using certificates or TCP for sending without encryption\n type: SSL\n chain: chain.crt # \\\n cert: .crt # Credential files downloaded from a Devo domain\n key: .key # /\ninputs:\n gcp:\n id: # The value of this field will be used internally for having independent persistence areas\n enabled: true # If \"false\" value this \"input\" will not be used by collector\n requests_per_second: 5 # Setup how many request API por second\n use_time_slots_for_retrievals: false\n autoconfig:\n enabled: true # Determines if the \"autoconfiguration\" functionality will be executed\n refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended.\n credentials:\n source_id: # This value will be used for adding to message \"tag\" as fourth level\n project_id: # This value will be used for adding to message \"tag\" as fifth level\n filename: # Credentials file generated from the \"GCP administration console\"\n file_content_base64: # Credentials file content generated from the \"GCP administration console\" in base64 format\n services:\n all_resources_na:\n sink_filter_resource: \"resource.type:\\\"\\\"\"\n sink_filter_resource_region: \"resource.type:\\\"\\\" AND resource.labels.zone:\\\"{region}\\\"\"\n regions:\n - northamerica\n - us\n all_resources_europe:\n sink_filter_resource: \"resource.type:\\\"\\\"\"\n sink_filter_resource_region: \"resource.type:\\\"\\\" AND resource.labels.zone:\\\"{region}\\\"\"\n regions:\n - europe ","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"The syntax to be used in ","type":"text"},{"text":"sink_filter_resource_*","type":"text","marks":[{"type":"strong"}]},{"text":" properties is detailed ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://cloud.google.com/logging/docs/view/advanced-queries"}}]},{"text":".","type":"text"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Custom ","type":"text","marks":[{"type":"strong"}]},{"text":"services, ","type":"text"},{"text":"autoconfig","type":"text","marks":[{"type":"strong"}]},{"text":" functionality ","type":"text"},{"text":"disabled ","type":"text","marks":[{"type":"strong"}]},{"text":"and using ","type":"text"},{"text":"regions","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"globals:\n debug: false\n id: not_used\n name: gcp\n persistence:\n type: filesystem # File system persistence ON\n config:\n directory_name: state # Directory where the persistence will be saved in case of using filesystem\noutputs:\n devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)\n type: devo_platform\n config:\n address: collector-eu.devo.io # URL to use for sending messages\n port: 443 # SSL for using certificates or TCP for sending without encryption\n type: SSL\n chain: chain.crt # \\\n cert: .crt # Credential files downloaded from a Devo domain\n key: .key # /\ninputs:\n gcp:\n id: # The value of this field will be used internally for having independent persistence areas\n enabled: true # If \"false\" value this \"input\" will not be used by collector\n requests_per_second: 5 # Setup how many request API por second\n use_time_slots_for_retrievals: false\n autoconfig:\n enabled: false # Determines if the \"autoconfiguration\" functionality will be executed\n refresh_interval_in_seconds: 3600 # Setting up the time for executing autoconfig interval. 3600 seconds is recommended.\n credentials:\n source_id: # This value will be used for adding to message \"tag\" as fourth level\n project_id: # This value will be used for adding to message \"tag\" as fifth level\n filename: # Credentials file generated from the \"GCP administration console\"\n file_content_base64: # Credentials file content generated from the \"GCP administration console\" in base64 format\n services:\n all_resources_na:\n sink_filter_resource: \"resource.type:\\\"\\\"\"\n sink_filter_resource_region: \"resource.type:\\\"\\\" AND resource.labels.zone:\\\"{region}\\\"\"\n regions:\n - northamerica\n - us\n all_resources_europe:\n sink_filter_resource: \"resource.type:\\\"\\\"\"\n sink_filter_resource_region: \"resource.type:\\\"\\\" AND resource.labels.zone:\\\"{region}\\\"\"\n regions:\n - europe","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"The syntax to be used in ","type":"text"},{"text":"sink_filter_resource_*","type":"text","marks":[{"type":"strong"}]},{"text":" properties is detailed ","type":"text"},{"text":"here","type":"text","marks":[{"type":"link","attrs":{"href":"https://cloud.google.com/logging/docs/view/advanced-queries"}}]},{"text":".","type":"text"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Custom ","type":"text","marks":[{"type":"strong"}]},{"text":"services and specifying \"","type":"text"},{"text":"subscription name","type":"text","marks":[{"type":"strong"}]},{"text":"\". In order to do this, the ","type":"text"},{"text":"autoconfig","type":"text","marks":[{"type":"strong"}]},{"text":" functionality must be ","type":"text"},{"text":"disabled","type":"text","marks":[{"type":"strong"}]},{"text":" and the use of the ","type":"text"},{"text":"regions","type":"text","marks":[{"type":"strong"}]},{"text":" section is ","type":"text"},{"text":"not allowed","type":"text","marks":[{"type":"strong"}]},{"text":" in this case (it will be ignored if exists):","type":"text"}]}]}]},{"type":"codeBlock","content":[{"text":"globals:\n debug: false\n id: not_used\n name: gcp\n persistence:\n type: filesystem # File system persistence ON\n config:\n directory_name: state # Directory where the persistence will be saved in case of using filesystem\noutputs:\n devo_1: # Cloud Devo config EU (for US use us.elb.relay.logtrust.net)\n type: devo_platform\n config:\n address: collector-eu.devo.io # URL to use for sending messages\n port: 443 # SSL for using certificates or TCP for sending without encryption\n type: SSL\n chain: chain.crt # \\\n cert: .crt # Credential files downloaded from a Devo domain\n key: .key # /\ninputs:\n gcp:\n id: # The value of this field will be used internally for having independent persistence areas\n enabled: true # If \"false\" value this \"input\" will not be used by collector\n requests_per_second: 5 # Setup how many request API por second\n use_time_slots_for_retrievals: false\n autoconfig:\n enabled: false # Determines if the \"autoconfiguration\" functionality will be executed\n refresh_interval_in_seconds: 90 # Setting up the time for executing autoconfig interval. 90sec is recommended.\n credentials:\n source_id: # This value will be used for adding to message \"tag\" as fourth level\n project_id: # This value will be used for adding to message \"tag\" as fifth level\n filename: # Credentials file generated from the \"GCP administration console\"\n file_content_base64: # Credentials file content generated from the \"GCP administration console\" in base64 format\n services:\n all_resources_usa:\n subscription_name: \"subscription-a\"\n all_resources_europe:\n subscription_name: \"subscription-b\"","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Download the Docker image","type":"text"}]},{"type":"paragraph","content":[{"text":"The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:","type":"text"}]},{"type":"table","attrs":{"layout":"default","localId":"8a0305e9-4a33-4c7e-8904-447bb6240e2e"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#c8e9f2","rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"Collector Docker image","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableHeader","attrs":{"colspan":1,"background":"#c8e9f2","rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"SHA-256 hash","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"collector-gcp-docker-image-1.1.2.tgz","type":"text","marks":[{"type":"link","attrs":{"href":"https://drive.google.com/file/d/1IvL6XFcvvuvQeyfSSx-5TKrJq6du77y0/view?usp=sharing"}}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[340.0]},"content":[{"type":"paragraph","content":[{"text":"0571ba2a9615f68efb4e5bc44197a33d3b6c28443b54173ace9c906f099c24fa","type":"text","marks":[{"type":"code"}]}]}]}]}]},{"type":"paragraph","content":[{"text":"Use the following command to add the Docker image to the system:","type":"text"}]},{"type":"codeBlock","content":[{"text":"gunzip -c collector-gcp-docker-image-.tgz | docker load","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"Once the Docker image is imported, it will show the real name of the Docker image (including version info).","type":"text"}]}]},{"type":"paragraph","content":[{"text":"The Docker image can be deployed on the following services:","type":"text"}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Docker","type":"text"}]},{"type":"paragraph","content":[{"text":"Execute the following command on the root directory ","type":"text"},{"text":"/devo-collectors/gcp/","type":"text","marks":[{"type":"code"}]}]},{"type":"codeBlock","content":[{"text":"docker run \\\n--name collector-gcp \\\n--volume $PWD/certs:/devo-collector/certs \\\n--volume $PWD/config:/devo-collector/config \\\n--volume $PWD/state:/devo-collector/state \\\n--env CONFIG_FILE=config-gcp.yaml \\\n--rm -it docker.devo.internal/collector/gcp:","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"Replace ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":" with the proper version.","type":"text"}]}]},{"type":"heading","attrs":{"level":3},"content":[{"text":"Docker Compose","type":"text"}]},{"type":"paragraph","content":[{"text":"The following Docker Compose file can be used to execute the Docker container. It must be created in the ","type":"text"},{"text":"/devo-collectors/gcp/ ","type":"text","marks":[{"type":"code"}]},{"text":"directory.","type":"text"}]},{"type":"codeBlock","content":[{"text":"version: '3'\nservices:\n collector-gcp:\n build:\n context: .\n dockerfile: Dockerfile\n image: docker.devo.internal/collector/gcp:${IMAGE_VERSION:-latest}\n container_name: collector-gcp\n volumes:\n - ./certs:/devo-collector/certs\n - ./config:/devo-collector/config\n - ./state:/devo-collector/state\n environment:\n - CONFIG_FILE=${CONFIG_FILE:-config-gcp.yaml}","type":"text"}]},{"type":"paragraph","content":[{"text":"To run the container using docker-compose, execute the following command from the ","type":"text"},{"text":"/devo-collectors/gcp/","type":"text","marks":[{"type":"code"}]},{"text":" directory:","type":"text"}]},{"type":"codeBlock","content":[{"text":"IMAGE_VERSION= docker-compose up -d","type":"text"}]},{"type":"paragraph"}]}],"version":1}

Browser not supported