Content Comparison

...

The information items that IntSights TIP provides are the Indicators of Compromise (IoC). Using the API, the collector extracts the IoC from IntSights and stores them in the Devo system as lookup tables, there are 5 types of IoC provided by IntSights: IP Address, DNS Domains, File Hashes, URLs, and Emails.

Data source description

Data source	Lookup	Collector service	Remote endpoint	Description
IP address	IntSights_IP_Address_IoC_List	iocs_list_ips	`https://api.intsights.com:443/public/`

v2

`v3/iocs?type[0]=IpAddresses`	IoC related to IP Address, stored using the IP as the primary key of the lookup
Domains	IntSights_Domain_IoC_List	iocs_list_domains	`https://api.intsights.com:443/public/`

v2

`v3/iocs?type[0]=Domains`	IoC related to Domains, stored using the DNS domain as the primary key of the lookup
File hashes	IntSights_Hash_IoC_List	iocs_list_hashes	`https://api.intsights.com:443/public/`

v2

`v3/iocs?type[0]=Hashes`	IoC related to File Hashes, stored using the hash value as the primary key of the lookup
URLs	IntSights_URL_IoC_List	iocs_list_urls	`https://api.intsights.com:443/public/`

v2

`v3/iocs?type[0]=Urls`	IoC related to URLs, stored using the URL as the primary key of the lookup
Email address	IntSights_Email_IoC_List	iocs_list_emails	`https://api.intsights.com:443/public/`

v2

v3/iocs?type[0]=Emails

IoC related to Email Addresses, stored using the email as the primary key of the lookup

Vendor setup

In order to configure the connection to IntSights, you need to generate a client_id and an api_key. These values are generated using the IntSights Cyber Intelligence asset in Rapid 7.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

...

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

...

Here are the steps to generate the credentials:

Action	Steps
Log in to the Rapid7 console: IntSights - Authentication	Log in to the Rapid7 IntSights console with your user credentials.
Generate API key	Click the Settings wheel icon in the left menu. Click the Subscription button. Click Generate API Key. If it already exists and you don't remember the API Key and it is not being used anywhere else, you can revoke it and create a new one. Copy the Account ID (parameter `client_id` in collector) and the API Key (`api_key` in collector).
Check permissions	Click the Settings wheel icon in the left menu. Click the Users button. You will see the administration page. Check that the permissions are correct.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Replace <version> with the required value.

-collector-intsights/ directory:

Rw ui tabs macro

Rw tab

title	Cloud collector

We use a piece of software called Collector Server to host and manage all our available collectors. If you want us to host this collector for you, get in touch with us and we will guide you through the configuration.

Rw tab

title	On-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the IntSights collector:

Code Block

<any_directory>
└── devo-collectors/
    └── devo-collector-intsights/
          ├── certs/
          │ ├── chain.crt
          │ ├── <your_domain>.key
          │ └── <your_domain>.crt
          └── config/
              └── config-intsight.yaml

Devo credentials

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in devo-collector-intsights/certs/. Learn more about security credentials in Devo here.

Image Removed

Image Added

Editing the config.yaml file

Code Block

globals:
  iddebug: <id<debug_collector>value>
  nameid: <namenot_collector>used
  debugname: <debug<name_value>collector>
  persistence:
    type: filesystem
    config:
      directory_name: state

 queue_max_size_in_mboutputs:
1024   queue_max_size_in_messagesdevo_1:
1000    queue_max_elapsed_time_in_sectype: 60devo_platform
  queue_wrap_max_size_in_messages: 100 
outputs:
  devo_1:
    type: devo_platform
    config:
      address: <devo_address>
      port: 443
      type: SSL
      chain: <chain_filename.crt>
      cert: <cert_filename.crt>
      key: <key_filename.key>

inputs:
  intsights:
    id: <input_id>
    enabled: <input_status>
    credentials:
      client_id: <client_id>
      api_key: <key_id>
    base_url: <base_url>
    services:
      iocs_list_ips:
        request_period_in_seconds: <request_period_in_seconds_ips> # Optional
        days_to_look_back: <days_to_look_back_ips>
      iocs_list_domains:
        request_period_in_seconds: <request_period_in_seconds_domains> # Optional
        days_to_look_back: <days_to_look_back_domains>
      iocs_list_hashes:
        request_period_in_seconds: <request_period_in_seconds_hashes> # Optional
        days_to_look_back: <days_to_look_back_hashes>
      iocs_list_urls:
        request_period_in_seconds: <request_period_in_seconds_urls> # Optional
        days_to_look_back: <days_to_look_back_urls>
      iocs_list_emails:
        request_period_in_seconds: <request_period_in_seconds_emails> # Optional
        days_to_look_back: <days_to_look_back_emails>

Replace the placeholders with your corresponding values:

Parameter	Data type	Type	Value range	Details
`id_collector`	int	Mandatory	Minimum length: 1 Maximum length: 5	Use this param to give a unique ID to this collector (not used)
`name_collector`	str	Mandatory	Minimum length: 1 Maximum length: 10	Use this param to give a valid name to this collector.
`debug_value`	bool	Mandatory	false / true	Use this param to enable or disable the debug logging level.
`devo_address`	str	Mandatory	collector-us.devo.io collector-eu.devo.io	Use this param to identify the Devo Cloud where the events will be sent.
`chain_filename`	str	Mandatory	Minimum length: 4 Maximum length: 20	Use this param to identify the chain.cert file downloaded from your Devo domain. Usually this file's name is: chain.crt
`cert_filename`	str	Mandatory	Minimum length: 4 Maximum length: 20	Use this param to identify the file.cert downloaded from your Devo domain.
`key_filename`	str	Mandatory	Minimum length: 4 Maximum length: 20	Use this param to identify the file.key downloaded from your Devo domain.
`input_id`	int	Mandatory	Minimum length: 1 Maximum length: 5	Use this param to give an unique id to this input service, (for instance 147 or 456)
`input_status`	bool	Mandatory	false / true	If the value is true, the input definition will be executed. If the value is false, the service will be ignored.
`client_id`	str	Mandatory	Minimum length: 1	This is the client_id given by Rapid7 IntSights
`api_key`	str	Mandatory	Minimum length: 1	This is the api_key given by Rapid7 IntSights
`base_url`	str	Mandatory	Regex: `^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.])([\/\w \.-])([a-z])$`	The URL of endpoint for IntSights API access. This value is given by Rapid7 console. A common value for base_url is usually https://api.intsights.com
`request_period_in_seconds`	int	Mandatory	Minimum value: 1800	Period in seconds used between each IoC pulling related data source. Use the following periods as reference: Run every 30 min → 1800 sec Run every 1 hour → 3600 sec Run every 6 hours → 21600 sec Run every 12 hours → 43200 sec Run every 24 hours → 86400 sec The IoC list changes very slowly, so it does not make sense to request it very frequently, in many installations it is enough to request it a few times a day. As the list is long, in any case a small request period (less than some minutes) is not advised.
`days_to_look_back`	int	Mandatory	Minimum value: 1 Maximum value: 5	The collector asks the API for all the IoCs from present moment to some days back, minimum in the last day (1 day back). Usual values for services: emails → 5 urls → 5 hashes → 5 domains → 5 ips → 2

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image	SHA-256 hash
collector-rapid7-intsights_intsights_ioc_collector_if-docker-image-12.12.0.tgz	`4a19987a2ad0bf02b1700c32aee2d5f0f47ed650d5a2d716898045e980d0eae91c96a75b0ed6b2002241c4fb68bc211d95e050f03c59c6e2ca47e37d19968f51`

Use the following command to add the Docker image to the system:

Code Block
gunzip -c collector-rapid7-intsights-docker-image-<version>.tgz \| docker load

Info
Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace "`<version>`" with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/devo-collector-intsights/

Code Block

docker run \
--name collector-rapid7-intsights\
--volume $PWD/certs:/devo-collector/certs \
--volume $PWD/config:/devo-collector/config \
--volume $PWD/state:/devo-collector/state \
--env CONFIG_FILE=config.yaml \
--rm -it docker.devo.internal/collectors/rapid7-intsights:<version>

Note
Replace `<version>` with the required value.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/devo-collector-intsights/ directory.

Code Block

version: '3'
services:
  intsights:
    build:
      context: .
      dockerfile: Dockerfile
    image: docker.devo.internal/collectors/rapid7-intsights:${IMAGE_VERSION:-latest}
    container_name: collector-rapid7-intsights
    volumes:
      - ./certs:/devo-collector/certs
      - ./config:/devo-collector/config
      - ./state:/devo-collector/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/devo-collector-intsights/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note

Code Block
IMAGE_VERSION=<version> docker-compose up -d

Note
Replace `<version>` with the required value.

Change log

Release

Released on

Release type

Details

Recommendations

v2.2.0

13 Oct 2023

Status

colour	Yellow
title	IMPROVEMENTS

Improvements:

Added status field to lookups
Parameter request_period_in_seconds made optional in config, default to 21600

Recommended Version

v2.1.0

02 Oct 2023

Status

colour	Red
title	BUG FIXING

Status

colour	Yellow
title	IMPROVEMENTS

Improvements

Upgraded DCSDK from 1.8.0 to 1.9.2
- Upgrade internal dependencies
- Store lookup instances into DevoSender to avoid creation of new instances for the same lookup
- Ensure service_config is a dict into templates
- Ensure special characters are properly sent to the platform
- Changed log level to some messages from info to debug
- Changed some wrong log messages
- Upgraded some internal dependencies
- Changed queue passed to setup instance constructor

Bug Fixing

Fixed format for input JSON data in case that they are a dict or list

Recommended Version

v2.0.0

11 Jul 2023

Status

colour	Red
title	BUG FIXING

Status

colour	Yellow
title	IMPROVEMENTS

Improvements

Upgraded DCSDK from 1.1.4 to 1.8.0
- Added log traces for knowing the execution environment status (debug mode)
- Fixes in the current puller template version
- Improved log trace details when runtime exceptions happen
- Refactored source code structure
- New "templates" functionality
- Functionality for detecting some system signals for starting the controlled stopping
- Input objects sends again the internal messages to devo.collectors.out table
- Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo
- Refactored source code structure
- Changed way of executing the controlled stopping
- Minimized probabilities of suffering a DevoSDK bug related to "sender" to be null
- Ability to validate collector setup and exit without pulling any data
- Ability to store in the persistence the messages that couldn't be sent after the collector stopped
- Ability to send messages from the persistence when the collector starts and before the puller begins working
- Ensure special characters are properly sent to the platform
- Added a lock to enhance sender object
- Added new class attrs to the __setstate__ and __getstate__ queue methods
- Fix sending attribute value to the __setstate__ and __getstate__ queue methods
- Added log traces when queues are full and have to wait
- Added log traces of queues time waiting every minute in debug mode
- Added method to calculate queue size in bytes
- Block incoming events in queues when there are no space left
- Send telemetry events to Devo platform
- Upgraded internal Python dependency Redis to v4.5.4
- Upgraded internal Python dependency DevoSDK to v5.1.3
- Fixed obfuscation not working when messages are sent from templates
- New method to figure out if a puller thread is stopping
- Upgraded internal Python dependency DevoSDK to v5.0.6
- Improved logging on messages/bytes sent to Devo platform
- Fixed wrong bytes size calculation for queues
- New functionality to count bytes sent to Devo Platform (shown in console log)
- Upgraded internal Python dependency DevoSDK to v5.0.4
- Fixed bug in persistence management process, related to persistence reset
- Aligned source code typing to be aligned with Python 3.9.x
- Inject environment property from user config
- Obfuscation service can be now configured from user config and module definiton
- Obfuscation service can now obfuscate items inside arrays

Bug Fixing

Endpoints have been changed from v2 to v3. v2 endpoints are no longer available in the API, therefore this collector can no longer be downgraded.

Recommended version

Highly recommended to update, the API no longer supports the endpoints used in previous versions.

Version	Old Version 2	New Version Current
Changes made by	Former user	Virginia Camuñas Núñez
Saved on	Jun 07, 2022	Feb 17, 2025

Content Comparison

Versions Compared

Key

Data source description

Vendor setup

Run the collector

Run the collector

Structure

Devo credentials

Editing the config.yaml file

Download the Docker image

Docker

Docker Compose

Change log

Improvements

Bug Fixing

Improvements

Bug Fixing