/
Authorize and start a new collector
Document toolboxDocument toolbox

Authorize and start a new collector

Owned by Former user (Deleted)
Last updated: Apr 17, 2025 by Laszlo Frazer
[ 1 Create a collector ]

Create a collector

To request data from a source, create a collector and provide it with the secrets it needs to make the request.

  1. Open the Cloud Collector App.

image-20250411-152809.png

  1. Select “Catalog.”

  2. Find the collector.

Contact us if you cannot find the collector in the catalog. Since some collectors are very flexible, it can be difficult to identify all the kinds of data they can collect. For example, the AWS SQS collector and Azure collector can collect all kinds of data from queues.

10_Start and authorize a new collector.png

  1. Select “Create instance.”

  2. Name the collector.

When naming the collector, ensure the name contains sufficient detail to streamline future secret rotation. This is crucial because expired secrets are a primary cause of collector failures.

  1. Edit the parameters.

    1. Change the id field to a unique five digit number in quotation marks. For example, if the time is 17:34:22, set the number to 17342 and it will probably be unique. If two collectors have the same id, data loss or corruption may occur.

20_Start and authorize a new collector.png

b. For information on how to fill out the other parameters, check the documentation for the specific collector.

c. Ensure the credentials fields are completed.

d. Leave the debug field set to false.

  1. The “Environment variables” are reserved for future development.

  2. Most customers will only need the default keychain. If special event load balancers have been purchased (for example, to meet complex data residency requirements) then uncheck the “Sender configuration” box and select the keychain associated with the correct event load balancer.

  3. Select “Create instance.”

image-20250411-152305.png

  1. The collector will take up to three minutes to start.

  2. To check the status of the collector:

    1. Get the collector ID from the “Installed collectors” page.

    2. Search for the collector ID in devo.collectors.out. An example query is

      from devo.collectors.out where startswith(hostname, "collector-563ae0ff0ba02ef4")

  3. If the data source has data available, go to “Data search” and click the refresh button to see which tables are being populated by all your collectors.

    40_Start and authorize a new collector.png

     

  4. If several collectors are using the same table, the hidden hostchain field can be used to identify them. For example,

    from my.app.test.writer group by split(hostchain,"-",1) as collector_id
50_Start and authorize a new collector.png

After creating a collector, it is helpful to set up monitoring.