Rapid7 insightvm Cloud Collector

[ 1 Devo collector features ] [ 2 Data sources ] [ 3 Accepted authentication methods ] [ 4 Minimum configuration required for basic pulling ] [ 5 Run the collector ] [ 6 Collector services detail ] [ 7 Collector operations ] [ 8 Change log ]

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (multipod)	`not allowed`
Running environments	`collector server` `On premise`
Populated Devo events	`table`
Flattening preprocessing	`no`

Data sources

Data source	Description	API endpoint	Collector service name	Devo table	Available from release

Data source	Description	API endpoint	Collector service name	Devo table	Available from release
Vulnerabilities	Get a list of Vulnerabilities .	`integration/vulnerabilities`	`vulnerabilities`	`vuln.rapid7.insightvm_cloud.vulnerability`	v1.0.0
Assets	Get a list of Assets.	`integration/assets`	`assets`	`vuln.rapid7.insightvm_cloud.asset`
Sites	Get a list of sites.	`integration/sites`	`sites`	`vuln.rapid7.insightvm_cloud.site`
Scans	Get a list of Scans.	`integration/scan`	`scans`	`vuln.rapid7.insightvm_cloud.scan`

For more information on how the events are parsed, visit our page.

Accepted authentication methods

api_key

required

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check running the collector section for details.

Setting	Details

Setting	Details
`api_key`	The `API Key`is your unique identifier for Rapid7 InsightVm Cloud.
`region`	The `region` is your region or Rapid7 InsightVm Cloud.
`api_version`	The `API Version` is the version of the api for Rapid7 InsightVm Cloud

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

2024-12-30T14:25:52.983    INFO InputProcess::MainThread -> InsightVMCloudPullerSetup(insightvm_cloud#12134,assets#predefined) -> Starting thread
2024-12-30T14:25:52.984    INFO InputProcess::MainThread -> InsightVMCloudPuller(insightvm_cloud#12134,assets#predefined) - Starting thread
2024-12-30T14:25:52.984    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-12-30T14:25:52.984    INFO OutputProcess::MainThread -> OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/state/865a79c1b99ad39b22becc235c9732cb"
2024-12-30T14:25:52.984 WARNING InputProcess::InsightVMCloudPullerSetup(insightvm_cloud#12134,assets#predefined) -> The token/header/authentication has not been created yet
2024-12-30T14:25:52.984    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/state/not_used/DevoSenderManager;internal_senders;devo_1.json.gz
2024-12-30T14:25:52.985    INFO InputProcess::MainThread -> InputMetricsThread -> Started thread for updating metrics values (update_period=10.0)
2024-12-30T14:25:52.985    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-12-30T14:25:52.985    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/state/e784c94a6fe28de5937e50674407d82e"
2024-12-30T14:25:52.987    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/state/not_used/DevoSender;internal_senders;devo_sender_0.json.gz
2024-12-30T14:25:52.988    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-12-30T14:25:52.988    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/state/4ff7b345dc444ac050cf75f93e5dcb3b"
2024-12-30T14:25:52.988    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Created persistence instance, filename_path: /home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/state/not_used/OutputInternalConsumer;internal_senders;0.json.gz
2024-12-30T14:25:52.989    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] There is no data persisted with the latest format, any previous persisted data will be migrated
2024-12-30T14:25:52.989    INFO OutputProcess::MainThread -> OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] No previous persistence file exists to migrate (Version 1), filename_path: "/home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/state/10dd360c86621afd5a28a029a0dddcf6"
2024-12-30T14:25:52.989    INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
2024-12-30T14:25:52.989    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
2024-12-30T14:25:52.990    INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
2024-12-30T14:25:52.990    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-30T14:25:52.990    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-30T14:25:52.990    INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
2024-12-30T14:25:52.991    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
2024-12-30T14:25:52.991    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-30T14:25:52.991    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-12-30T14:25:52.991    INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
2024-12-30T14:25:52.991    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-30T14:25:52.992    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-30T14:25:52.992    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-12-30T14:25:52.992    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
2024-12-30T14:25:52.993    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
2024-12-30T14:25:52.993    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-30T14:25:52.993    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-12-30T14:25:52.993    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-30T14:25:52.994    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-30T14:25:52.994    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-12-30T14:25:52.994    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-30T14:25:52.994    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-12-30T14:25:52.994    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-30T14:25:52.994    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-30T14:25:52.994    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_1) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-12-30T14:25:52.994    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-30T14:25:52.994    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-12-30T14:25:52.995    INFO OutputProcess::MainThread -> OutputMetricsThread -> Started thread for updating metrics values (update_period=10.0)
2024-12-30T14:25:53.009    INFO InputProcess::MainThread -> [GC] global: 32.3% -> 32.4%, process: RSS(62.12MiB -> 62.62MiB), VMS(521.70MiB -> 521.70MiB)
2024-12-30T14:25:53.012    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "vendor_requests" created: "Number of requests received from the vendor API", unit: "requests"
2024-12-30T14:25:53.012    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_received" created: "Number of messages received from the vendor API", unit: "1"
2024-12-30T14:25:53.012    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_removed" created: "Number of messages removed by the collector", unit: "1"
2024-12-30T14:25:53.012    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_filtered" created: "Number of messages filtered by the collector", unit: "1"
2024-12-30T14:25:53.013    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_counter" created: "Number of messages enqueued", unit: "1"
2024-12-30T14:25:53.013    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_bytes" created: "Number of bytes enqueued", unit: "1"
2024-12-30T14:25:53.013    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_counter" created: "Number of messages enqueued", unit: "1"
2024-12-30T14:25:53.013    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_bytes" created: "Number of messages enqueued", unit: "1"
2024-12-30T14:25:53.013    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_counter" created: "Number of messages enqueued in the queue", unit: "1"
2024-12-30T14:25:53.014    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_bytes" created: "Number of messages enqueued in the queue", unit: "1"
2024-12-30T14:25:53.014    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Gauge "module_global_status" created: "Global status of current module", unit: "1"
2024-12-30T14:25:53.021    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_counter" created: "Number of messages sent to the defined output", unit: "1"
2024-12-30T14:25:53.022    INFO OutputProcess::MainThread -> [GC] global: 32.4% -> 32.4%, process: RSS(61.73MiB -> 62.61MiB), VMS(1.07GiB -> 1.07GiB)
2024-12-30T14:25:53.022    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_bytes" created: "Number of bytes sent to the defined output", unit: "1"
2024-12-30T14:25:53.740    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/certs/chain.crt", "cert_path": "/home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/certs/int-if-integrations-india.crt", "key_path": "/home/md_tausif/gitlab/devo-collector-rapid7-insightvm-cloud/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-apac-0046", session_id: "137264546097760"
2024-12-30T14:25:53.740    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-30T14:25:55.171    INFO InputProcess::InsightVMCloudPullerSetup(insightvm_cloud#12134,assets#predefined) -> Setup for module <InsightVMCloudPuller> has been successfully executed

Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in an organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-12-30T13:26:49.390    INFO InputProcess::InsightVMCloudPuller(insightvm_cloud#12134,scans#predefined) -> Pull Started
2024-12-30T13:26:49.390    INFO InputProcess::InsightVMCloudPuller(insightvm_cloud#12134,scans#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1735545289387):Number of requests made: 1; Number of events received: 100; Number of duplicated events filtered out: 0; Number of events generated and sent: 100; Average of events per second: 209.336.
2024-12-30T13:26:49.390    INFO InputProcess::InsightVMCloudPuller(insightvm_cloud#12134,scans#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1735545289387):Number of requests made: 1; Number of events received: 100; Number of duplicated events filtered out: 0; Number of events generated and sent: 100; Average of events per second: 209.336.
2024-12-30T13:26:49.390    INFO InputProcess::InsightVMCloudPuller(insightvm_cloud#12134,scans#predefined) -> The data is up to date!

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-12-30T13:26:49.390    INFO InputProcess::InsightVMCloudPuller(insightvm_cloud#12134,scans#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1735545289387):Number of requests made: 1; Number of events received: 100; Number of duplicated events filtered out: 0; Number of events generated and sent: 100; Average of events per second: 209.336.

Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in an organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in an organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in an organized way and delivering the events via SDK.

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

Collector operations

Change log

Release	Released on	Release type	Recommendations

Release	Released on	Release type	Recommendations
`v1.1.0`	Jan 6, 2025	Imrovements	`Recommended version`
`v1.1.0`
`v1.0.0`	Mar 22, 2024	Initial release	`Updated`