Rapid7 IntSights collector

[ 1 Service description ] [ 2 Data source description ] [ 3 Vendor setup ] [ 4 Run the collector ] [ 5 Change log ]

Service description

The Rapid7 IntSights collector ingests threat indicators from the Insights Threat Intelligence Platform (TIP) as Devo lookup tables. This allows the use of the indicators as a correlation source when using the Devo platform to analyze security data from other systems as part of reactive alerting and proactive threat hunting.

IntSights (a Rapid 7 company) is a security company specialized in Endpoint Security and threat detection. IntSights provides cloud-native external threat detection to further extend Rapid7’s security operations platform, providing customers with end-to-end external and internal threat detection, automation, and remediation.

The information items that IntSights TIP provides are the Indicators of Compromise (IoC). Using the API, the collector extracts the IoC from IntSights and stores them in the Devo system as lookup tables, there are 5 types of IoC provided by IntSights: IP Address, DNS Domains, File Hashes, URLs, and Emails.

Data source description

Data source	Lookup	Collector service	Remote endpoint	Description

Data source	Lookup	Collector service	Remote endpoint	Description
IP address	IntSights_IP_Address_IoC_List	iocs_list_ips	`https://api.intsights.com:443/public/v3/iocs?type[0]=IpAddresses`	IoC related to IP Address, stored using the IP as the primary key of the lookup
Domains	IntSights_Domain_IoC_List	iocs_list_domains	`https://api.intsights.com:443/public/v3/iocs?type[0]=Domains`	IoC related to Domains, stored using the DNS domain as the primary key of the lookup
File hashes	IntSights_Hash_IoC_List	iocs_list_hashes	`https://api.intsights.com:443/public/v3/iocs?type[0]=Hashes`	IoC related to File Hashes, stored using the hash value as the primary key of the lookup
URLs	IntSights_URL_IoC_List	iocs_list_urls	`https://api.intsights.com:443/public/v3/iocs?type[0]=Urls`	IoC related to URLs, stored using the URL as the primary key of the lookup
Email address	IntSights_Email_IoC_List	iocs_list_emails	`https://api.intsights.com:443/public/v3/iocs?type[0]=Emails`	IoC related to Email Addresses, stored using the email as the primary key of the lookup

Vendor setup

In order to configure the connection to IntSights, you need to generate a client_id and an api_key. Here are the steps to generate the credentials:

Action	Steps

Action	Steps
Log in to the Rapid7 console: IntSights - Authentication	Log in to the Rapid7 IntSights console with your user credentials.
Generate API key	Click the Settings wheel icon in the left menu. Click the Subscription button. Click Generate API Key. If it already exists and you don't remember the API Key and it is not being used anywhere else, you can revoke it and create a new one. Copy the Account ID (parameter `client_id` in collector) and the API Key (`api_key` in collector).
Check permissions	Click the Settings wheel icon in the left menu. Click the Users button. You will see the administration page. Check that the permissions are correct.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Change log

Release	Released on	Release type	Details	Recommendations

Release

Released on

Release type

Details

Recommendations

v2.2.0

Oct 13, 2023

IMPROVEMENTS

Improvements:

Added status field to lookups
Parameter request_period_in_seconds made optional in config, default to 21600

Recommended Version

v2.1.0

Oct 2, 2023

BUG FIXING IMPROVEMENTS

Improvements

Upgraded DCSDK from 1.8.0 to 1.9.2
- Upgrade internal dependencies
- Store lookup instances into DevoSender to avoid creation of new instances for the same lookup
- Ensure service_config is a dict into templates
- Ensure special characters are properly sent to the platform
- Changed log level to some messages from info to debug
- Changed some wrong log messages
- Upgraded some internal dependencies
- Changed queue passed to setup instance constructor

Bug Fixing

Fixed format for input JSON data in case that they are a dict or list

Recommended Version

v2.0.0

Jul 11, 2023

BUG FIXING IMPROVEMENTS

Improvements

Upgraded DCSDK from 1.1.4 to 1.8.0
- Added log traces for knowing the execution environment status (debug mode)
- Fixes in the current puller template version
- Improved log trace details when runtime exceptions happen
- Refactored source code structure
- New "templates" functionality
- Functionality for detecting some system signals for starting the controlled stopping
- Input objects sends again the internal messages to devo.collectors.out table
- Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo
- Refactored source code structure
- Changed way of executing the controlled stopping
- Minimized probabilities of suffering a DevoSDK bug related to "sender" to be null
- Ability to validate collector setup and exit without pulling any data
- Ability to store in the persistence the messages that couldn't be sent after the collector stopped
- Ability to send messages from the persistence when the collector starts and before the puller begins working
- Ensure special characters are properly sent to the platform
- Added a lock to enhance sender object
- Added new class attrs to the __setstate__ and __getstate__ queue methods
- Fix sending attribute value to the __setstate__ and __getstate__ queue methods
- Added log traces when queues are full and have to wait
- Added log traces of queues time waiting every minute in debug mode
- Added method to calculate queue size in bytes
- Block incoming events in queues when there are no space left
- Send telemetry events to Devo platform
- Upgraded internal Python dependency Redis to v4.5.4
- Upgraded internal Python dependency DevoSDK to v5.1.3
- Fixed obfuscation not working when messages are sent from templates
- New method to figure out if a puller thread is stopping
- Upgraded internal Python dependency DevoSDK to v5.0.6
- Improved logging on messages/bytes sent to Devo platform
- Fixed wrong bytes size calculation for queues
- New functionality to count bytes sent to Devo Platform (shown in console log)
- Upgraded internal Python dependency DevoSDK to v5.0.4
- Fixed bug in persistence management process, related to persistence reset
- Aligned source code typing to be aligned with Python 3.9.x
- Inject environment property from user config
- Obfuscation service can be now configured from user config and module definiton
- Obfuscation service can now obfuscate items inside arrays

Bug Fixing

Endpoints have been changed from v2 to v3. v2 endpoints are no longer available in the API, therefore this collector can no longer be downgraded.

Recommended version

Highly recommended to update, the API no longer supports the endpoints used in previous versions.