Document toolboxDocument toolbox

AlienVault OTX Pulse collector

Overview

AlienVault OTX provides open access to a global community of threat researchers and security professionals. It delivers community-generated threat data, enables collaborative research, and automates the process of updating your security infrastructure with threat data from any source. OTX enables anyone in the security community to actively discuss, research, and share the latest threat data, trends, and techniques, strengthening your defenses while helping others do the same.

This document provides information about the AlienVault-OTX Pulse Collector, which facilitates automated interactions, with an AlienVault-OTX server to perform operations, such as retrieving details for an indicator, and for a pulse.

AlienVault OTX Pulse

Pulses are the format for the OTX community to share information about threats. Pulses provide you with a summary of the threat, a view into the software targeted, and the related indicators of compromise (IOC) that can be used to detect the threats. A pulse consists of at least one, but more often multiple Indicators of Compromise (IoCs). An IoC is an artifact observed on a network or in an endpoint judged with a high degree of confidence to be a threat vector. Examples of threat vectors include campaigns or infrastructures used by an attacker.

The table in this article provides a list of IoC types.

Data source description

Data source

Description

API endpoint

Collector service

Devo table

Data source

Description

API endpoint

Collector service

Devo table

Pulses

Threat intelligence subscriptions

  • All pulses by users you are subscribed to

  • All pulses you are directly subscribed to

  • All pulses you have created yourself

  • All pulses from groups you are a member of

/api/v1/pulses/subscribed

alienvault_otx

threatintel.alienvault_otx.pulses.indicators

The data retrieved from AlientVault OTX is adapted to be stored in Devo. Each received pulse contains several hundred or thousands of indicators. Each indicator is stored individually in the Devo table threatintel.alienvault_otx.indicators, but combined with its pulse information.

An example of the information item stored in Devo, containing both indicator and pulse data combined, can be seen in Example 1:

{"eventdate":"2022-03-31T10:40:09.509+0200","hostname":"2020-hostname", "id":"32321","indicator":"malware.indicator.web","type":"domain","created":"2022-03-31T10:02:54.000+0200","content":"","title":"","description":"", "expiration":"","is_active":"1","role":"malware_hosting", "pulse_id":"1ff3a9a","pulse_name":"New Wave Of Phishing Campaign","pulse_description":"","pulse_author_name":"AlienVault", "pulse_modified":"2022-03-31T10:02:52.294+0200","pulse_created":"2022-03-31T10:02:52.294+0200", "pulse_revision":"1","pulse_tlp":"white","pulse_public":"1","pulse_adversary":"", "pulse_tags":"["phishing", "trojan"]","pulse_targeted_countries":"[]","pulse_malware_families":"["Family 1"]","pulse_attack_ids":"["T1XX1","TA0XX2"]", "pulse_references":"[\"https://reference.web\"]","pulse_industries":"[\"Finance\"]","pulse_extract_source":"[]"}

Example 2

{"eventdate":"2022-03-31T10:39:23.475+0200","hostname":"2020-hostname", "id":"550876","indicator":"2.3.4.5","type":"IPv4","created":"2022-03-12T12:18:02.000+0100","content":"","title":"Blaster UDP, Trojan from scan.example.org port 48909","description":"", "expiration":"2022-04-11T13:00:00.000+0200","is_active":"1","role":"trojan", "pulse_id":"606d75c1189a9430","pulse_name":"Example Honeypot","pulse_description":"Honeypot","pulse_author_name":"john", "pulse_modified":"2022-03-31T10:39:02.654+0200","pulse_created":"2021-04-07T11:05:05.353+0200", "pulse_revision":"1","pulse_tlp":"white","pulse_public":"1","pulse_adversary":"", "pulse_tags":"["honeypot", "rdp", "ssh"]","pulse_targeted_countries":"["Italy"]","pulse_malware_families":"[]","pulse_attack_ids":"[]", "pulse_references":"[]","pulse_industries":"[]","pulse_extract_source":"[]"}

Vendor setup

In order to enable AlienVault OTX Pulse Devo Collector, you will need to follow the following steps:

  1. Go to AlienVault Open Threat Exchange.

  2. Click on the Login tab. If you don’t have an account you can sign up, and you will be able to access detailed documentation as well as your API key via the dashboard.

  3. Type your login ID and password (provided by AlienVault) and log in. 

  4. After the console loads, click API Integration on the menu. 

  5. This loads the DirectConnect API page. This section of the panel is also where you’ll be able to confirm from the OTX side that your connection is functional.

  6. Copy your OTX API key. This is the value that you need to update the api_key value in your collector configuration file.

Run the collector

Change log

Release

Released on

Release t

ype

Details

Recommendations

Release

Released on

Release t

ype

Details

Recommendations

v1.1.0

Aug 25, 2023

IMPROVEMENT

Improvements

  • Upgraded DCSDK from 1.1.4 to 1.9.1

    • Store lookup instances into DevoSender to avoid creation of new instances for the same lookup

    • Ensure service_config is a dict into templates

    • Ensure special characters are properly sent to the platform

    • Changed log level to some messages from info to debug

    • Changed some wrong log messages

    • Upgraded some internal dependencies

    • Changed queue passed to setup instance constructor

    • Added log traces for knowing the execution environment status (debug mode)

    • Fixes in the current puller template version

    • Improved log trace details when runtime exceptions happen

    • Refactored source code structure

    • New “templates” functionality

    • Functionality for detecting some system signals for starting the controlled stopping

    • Input objects sends again the internal messages to devo.collectors.out table

    • Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo

    • Refactored source code structure

    • Changed way of executing the controlled stopping

    • Minimized probabilities of suffering a DevoSDK bug related to “sender” to be null

    • Ability to validate collector setup and exit without pulling any data

    • Ability to store in the persistence the messages that couldn’t be sent after the collector stopped

    • Ability to send messages from the persistence when the collector starts and before the puller begins working

    • Ensure special characters are properly sent to the platform

    • Added a lock to enhance sender object

    • Added new class attrs to the setstate and getstate queue methods

    • Fix sending attribute value to the setstate and getstate queue methods

    • Added log traces when queues are full and have to wait

    • Added log traces of queues time waiting every minute in debug mode

    • Added method to calculate queue size in bytes

    • Block incoming events in queues when there are no space left

    • Send telemetry events to Devo platform

    • Upgraded internal Python dependency Redis to v4.5.4

    • Upgraded internal Python dependency DevoSDK to v5.1.3

    • Fixed obfuscation not working when messages are sent from templates

    • New method to figure out if a puller thread is stopping

    • Upgraded internal Python dependency DevoSDK to v5.0.6

    • Improved logging on messages/bytes sent to Devo platform

    • Fixed wrong bytes size calculation for queues

    • New functionality to count bytes sent to Devo Platform (shown in console log)

    • Upgraded internal Python dependency DevoSDK to v5.0.4

    • Fixed bug in persistence management process, related to persistence reset

    • Aligned source code typing to be aligned with Python 3.9.x

    • Inject environment property from user config

    • Obfuscation service can be now configured from user config and module definitonStore lookup instances into DevoSender to avoid creation of new instances for the same lookup

    • Ensure service_config is a dict into templates

    • Ensure special characters are properly sent to the platform

    • Changed log level to some messages from info to debug

    • Changed some wrong log messages

    • Upgraded some internal dependencies

    • Changed queue passed to setup instance constructor

    • Added log traces for knowing the execution environment status (debug mode)

    • Fixes in the current puller template version

    • Improved log trace details when runtime exceptions happen

    • Refactored source code structure

    • New “templates” functionality

    • Functionality for detecting some system signals for starting the controlled stopping

    • Input objects sends again the internal messages to devo.collectors.out table

    • Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo

    • Refactored source code structure

    • Changed way of executing the controlled stopping

    • Minimized probabilities of suffering a DevoSDK bug related to “sender” to be null

    • Ability to validate collector setup and exit without pulling any data

    • Ability to store in the persistence the messages that couldn’t be sent after the collector stopped

    • Ability to send messages from the persistence when the collector starts and before the puller begins working

    • Ensure special characters are properly sent to the platform

    • Added a lock to enhance sender object

    • Added new class attrs to the setstate and getstate queue methods

    • Fix sending attribute value to the setstate and getstate queue methods

    • Added log traces when queues are full and have to wait

    • Added log traces of queues time waiting every minute in debug mode

    • Added method to calculate queue size in bytes

    • Block incoming events in queues when there are no space left

    • Send telemetry events to Devo platform

    • Upgraded internal Python dependency Redis to v4.5.4

    • Upgraded internal Python dependency DevoSDK to v5.1.3

    • Fixed obfuscation not working when messages are sent from templates

    • New method to figure out if a puller thread is stopping

    • Upgraded internal Python dependency DevoSDK to v5.0.6

    • Improved logging on messages/bytes sent to Devo platform

    • Fixed wrong bytes size calculation for queues

    • New functionality to count bytes sent to Devo Platform (shown in console log)

    • Upgraded internal Python dependency DevoSDK to v5.0.4

    • Fixed bug in persistence management process, related to persistence reset

    • Aligned source code typing to be aligned with Python 3.9.x

    • Inject environment property from user config

    • Obfuscation service can be now configured from user config and module definiton

Recommended version