Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Rw ui tabs macro
Rw tab
titleCloud collector

We use a piece of software called Collector Server to host and manage all our available collectors.

To enable the collector for a customer:

  1. In the Collector Server GUI, access the domain in which you want this instance to be created

  2. Click Add Collector and find the one you wish to add.

  3. In the Version field, select the latest value.

  4. In the Collector Name field, set the value you prefer (this name must be unique inside the same Collector Server domain).

  5. In the sending method select Direct Send. Direct Send configuration is optional for collectors that create Table events, but mandatory for those that create Lookups.

  6. In the Parameters section, establish the Collector Parameters as follows below:

Editing the JSON configuration

Code Block
{
  "cisco_meraki": {
    "id": "<short_unique_identifier>",
    "enabled": true,
    "requests_per_second": 5,
    "services": {
      "network-events": {
        "api_key": "<api_key>",
        "start_time": "2021-01-01T00:00:00.000000Z"
      },
      "security-events": {
        "api_key": "<api_key>",
        "start_time": "2021-01-01T00:00:00.000000Z"
      },
      "changelog": {
        "api_key": "<api_key>",
        "start_time": "2021-01-01T00:00:00.000000Z"
      }
    }
  }
}
Info

Replace the placeholders <short_unique_identifier> and <api_key> with the proper values (obtained in previous sections of this document, except the <short_unique_identifier> valuethat can have the value you choose).

The value chosen for the id field will be used internally for having independent persistence areas.

Cisco meraki_10.png
Rw tab
titleOn-premise collector

This data collector can be run in any machine that has the Docker service available because it should be executed as a docker container. The following sections explain how to prepare all the required setup for having the data collector running.

Structure

The following directory structure should be created for being used when running the Cisco Meraki collector:

Code Block
<any_directory>
└── devo-collectors/
    └── cisco_meraki/
        ├── certs/
        │   ├── chain.crt
        │   ├── <your_domain>.key
        │   └── <your_domain>.crt
        └── config/ 
            └── config-cisco_meraki.yaml

Devo certificates

In Devo, go to Administration → Credentials → X.509 Certificates, download the Certificate, Private key and Chain CA and save them in <any directory>/devo-collectors/cisco_meraki/certs. Learn more about security credentials in Devo here.

Editing the config-cisco_meraki.yaml file

In the config-cisco_meraki.yaml file, replace the <short_unique_identifier>, <api_key> values and enter the ones that retrieved in the previous steps. In the <short_unique_identifier> placeholder, enter the chosen value.

Code Block
globals:
  debug: false
  id: not_used
  name: cisco_meraki
  persistence:
    type: filesystem                                           # File system persistence ON
    config:
      directory_name: state                                    # Directory where the persistence will be saved in case of using filesystem
outputs:
  devo_1:                                                      # Cloud Devo config EU (for US use collector-us.devo.io)
    type: devo_platform
    config:        
      address: eu.elb.relay.logtrust.net
      port: 443
      type: SSL
      chain: chain.crt
      cert: <your_domain>.crt
      key: <your_domain>.key
inputs:
  cisco_meraki:
    id: <short_unique_identifier>                              # The value of this field will be used internally for having independent persistence areas
    enabled: true
    requests_per_second: 5                                     # Setup how many request API por second
    services:                                                  # Services available for this collector are Alerts, Secure Score and Secure score control profile
      network-events:
        api_key: '<api_key>'                                   # API Key obtained in the Meraki profile
        start_time: '2021-01-01T00:00:00.000000Z'              # Collector Initial time.
      security-events:
        api_key: '<api_key>'                                   # API Key obtained in the Meraki profile
        start_time: '2021-01-01T00:00:00.000000Z'              # Collector Initial time.
      changelog:
        api_key: '<api_key>'                                   # API Key obtained in the Meraki profile
        start_time: '2021-01-01T00:00:00.000000Z'              # Collector Initial time.
Note

The “start_time” fields must have the following format:

Alerts “start_time” format: 0000-00-00T00:00:00.000Z

Note

The Security-events Service may generate error logs if you do not have an MX appliance.

Download the Docker image

The collector should be deployed as a Docker container. Download the Docker image of the collector as a .tgz file by clicking the link in the following table:

Collector Docker image

SHA-256 hash

collector-meraki_collector-docker-image-1.7.0

d5cf71ca38ad3a2a270c00bb617e9c53b53ca94ce3abd00c5145194f08406c6c

Use the following command to add the Docker image to the system:

Code Block
gunzip -c collector-cisco-meraki-docker-image-<version>.tgz | docker load

Once the Docker image is imported, it will show the real name of the Docker image (including version info). Replace "<version>" with a proper value.

The Docker image can be deployed on the following services:

Docker

Execute the following command on the root directory <any_directory>/devo-collectors/cisco_meraki/

Code Block
docker run \
--name collector-cisco_meraki \
--volume $PWD/certs:/devo-collector/cisco_meraki/certs \
--volume $PWD/config:/devo-collector/cisco_meraki/config \
--volume $PWD/state:/devo-collector/cisco_meraki/state \
--env CONFIG_FILE=config-cisco_meraki.yaml \
--rm -it docker.devo.internal/collector/meraki_collector:<version>
Note

Replace <version> with the required value.

Docker Compose

The following Docker Compose file can be used to execute the Docker container. It must be created in the <any_directory>/devo-collectors/cisco_meraki/ directory.

Code Block
version: '3'
services:
  collector-cisco_meraki:
    image: docker.devo.internal/collector/meraki_collector:${IMAGE_VERSION:-latest}
    container_name: collector-cisco_meraki
    volumes:
      - ./certs:/devo-collector/cisco_meraki/certs
      - ./config:/devo-collector/cisco_meraki/config
      - ./state:/devo-collector/cisco_meraki/state
    environment:
      - CONFIG_FILE=${CONFIG_FILE:-config-cisco_meraki.yaml}

To run the container using docker-compose, execute the following command from the <any_directory>/devo-collectors/cisco_meraki/ directory:

Code Block
IMAGE_VERSION=<version> docker-compose up -d
Note

Replace <version> with the required value.

Activeboards

A In Exchange a number of predefined dashboards that make use of the configured collectors can be downloaded here. To instantiate them, follow these instructions:

...

Create a new Devo Activeboard in your domain.

...

In edit mode, click on the ellipsis button and select Edit raw configuration.

...

Open the downloaded file, select all the text, and copy it to the clipboard.

...

Paste the contents of the file in the raw editor. Make sure you replace completely the existing configuration.

...

installed directly in your domain to start working with your data.

Change log

Release

Released on

Release type

Details

Recommendations

v1.7.0

Status
colourGreen
titleIMPROVEMENTS
Status
colourRed
titleBUG FIX

Bug fixes:

  • Changed the api_mode to false in metadata file.

Improvements:

  • Update DCSDK from 1.11.1 to 1.12.2

    • Added new sender for relay in house + TLS

    • Added persistence functionality for gzip sending buffer

    • Added Automatic activation of gzip sending

    • Improved behaviour when persistence fails

    • Upgraded DevoSDK dependency

    • Fixed console log encoding

    • Restructured python classes

    • Improved behavior with non-utf8 characters

    • Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

    • New persistence format/structure (compression in some cases)

    • Removed dmesg execution (It was invalid for docker execution)

    • DevoSDK has been updated to version 5.4.0

  • Updated the docker base image to 1.3.0

Recommended version

v1.6.0

Status
colourGreen
titleIMPROVEMENTS
Status
colourRed
titleBUG FIX

Improvements:

  1. Update DCSDK from 1.10.2 to 1.11.1

    • Updated DevoSDK to v5.1.10

    • Fix for SyslogSender related to UTF-8

    • Enhace of troubleshooting. Trace Standardization, Some traces has been introduced.

    • Introduced a mechanism to detect "Out of Memory killer" situation.

    • Updated DevoSDK to v5.1.9.

    • Fixed some bug related to development on MacOS.

    • Added an extra validation and fix when the DCSDK receives a wrong timestamp format.

    • Added an optional config property for use the Syslog timestamp format in a strict way.

Bug fixing:

  • Programmatically updated the start date if more than 7 days old.

Upgrade

v1.5.1

Status
colourGreen
titleIMPROVEMENTS
Status
colourRed
titleBUG FIX

Improvements:

  1. Update DCSDK from 1.9.0 to 1.10.2

    • Changed log level to some messages from info to debug

    • Changed some wrong log messages

    • Upgraded some internal dependencies

    • Changed queue passed to setup instance constructor

    • Ability to validate collector setup and exit without pulling any data

    • Ability to store in the persistence the messages that couldn't be sent after the collector stopped

    • Ability to send messages from the persistence when the collector starts and before the puller begins working

    • Ensure special characters are properly sent to the platform

Bug fixing:

  • Fixed the API parameters to get the correct data to avoid repeated API calls and hence the delay in security events.

Upgrade

v1.4.0

Status
colourGreen
titleIMPROVEMENTS
Status
colourRed
titleBUG FIX

Improvements:

  1. Update DCSDK from 1.1.4 to 1.9.0

    • Ability to validate collector setup and exit without pulling any data

    • Ability to store in the persistence the messages that couldn't be sent after the collector stopped

    • Ability to send messages from the persistence when the collector starts and before the puller begins working

    • Ensure special characters are properly sent to the platform

    • Added a lock to enhance sender object

    • Added new class attrs to the setstate and getstate queue methods

    • Fix sending attribute value to the setstate and getstate queue methods

    • Added log traces when queues are full and have to wait

    • Added log traces of queues time waiting every minute in debug mode

    • Added method to calculate queue size in bytes

    • Block incoming events in queues when there are no space left

    • Send telemetry events to Devo platform

    • Upgraded internal Python dependency Redis to v4.5.4

    • Upgraded internal Python dependency DevoSDK to v5.1.3

    • Fixed obfuscation not working when messages are sent from templates

    • New method to figure out if a puller thread is stopping

    • Upgraded internal Python dependency DevoSDK to v5.0.6

    • Improved logging on messages/bytes sent to Devo platform

    • Fixed wrong bytes size calculation for queues

    • New functionality to count bytes sent to Devo Platform (shown in console log)

    • Upgraded internal Python dependency DevoSDK to v5.0.4

    • Fixed bug in persistence management process, related to persistence reset

    • Aligned source code typing to be aligned with Python 3.9.x

    • Inject environment property from user config

    • Obfuscation service can be now configured from user config and module definiton

    • Obfuscation service can now obfuscate items inside arrays

Bug fixing:

  • Updated the limits of the API: The limits have been modified with the official values. This fixes throttling issues.

Upgrade

v1.3.7

Status
colourRed
titleBUG FIX

Bug fixing:

  • Fixed a bug in the security-events service (MX events) when no new events are received.

Upgrade

v1.3.6

Status
colourGreen
titleIMPROVEMENTS
Status
colourRed
titleBUG FIX

Improvements:

  • The number of debug traces has been increased to provide better visibility when troubleshooting.

  • The network_security_events (MX) service has been enhanced with new logic that avoids requesting MX events from networks without MX appliances. This reduces the number of unnecessary API requests to Meraki that were returning 400 HTTP CODE.

  • Meraki Python package has been upgraded from v1.18.2 to v1.22.1

  • The events are sent to Devo in batches, increasing the performance.

Bug fixing:

  • Fixed a bug where the data extraction services via network_event_log and network_security_events stopped pulling events after retrieving the first page of 1,000 events. This behavior was causing some delay in the ingest in networks with a high volume of events.

  • Fixed how persistence of the network_security_events service is handled and it now stores a unique save point for each available network instead of one for all networks.

  • Fixed a bug where events were being sent to Devo without proper ordering. Now all events are ordered from the origin by the API.

Upgrade

v1.3.5

Status
colourGreen
titleIMPROVEMENTS
Status
colourRed
titleBUG FIX

Improvements:

  • The number of debug traces has been increased to provide better visibility when troubleshooting.

Bug fixes:

  • Logging AssertionError trace thrown when requesting invalid product types from networks without those product types is now logged in debug level.

Upgrade

v1.3.4

Status
colourGreen
titleIMPROVEMENTS
Status
colourRed
titleBUG FIX

Improvements:

  • When the Meraki API returns an HTTP CODE 429 (Too many requests), the collector handle it to avoid overflooding.

Bug fixes:

  • Some error messages were logged at the debug level and are now logged at the error level correctly.

Upgrade

v1.3.2

Status
colourYellow
titleVULNS

Vulnerabilities mitigation:

  • All critical and high vulnerabilities have been mitigated.

Upgrade

v1.3.0

Status
colourPurple
titleFEATURE
Status
colourYellow
titleVULNS
Status
colourGreen
titleIMPROVEMENTS

New features:

  • The resilience has been improved with a new feature that restart the collector when the Devo connections is lost and it cannot be recovered.

Improvements:

  • The underlay IFC SDK has been updated to v1.1.3.

Vulnerabilities mitigation:

  • All critical and high vulnerabilities have been mitigated.

Upgrade

...