Release notes for Azure collector

Features

New autocategorization rules for

• cloud.azure.ah.alert_info

• cloud.azure.ah.alert_evidence

• cloud.azure.sql.securityauditevents

• cloud.azure.vm.subassessment

• cloud.azure.virtualnetwork.net_sec_group_event

• cloud.azure.eh.metrics

• cloud.azure.firewall.network_rule

• cloud.azure.firewall.application_rule

• cloud.azure.firewall.dns_query

• cloud.azure.storage.storageread

• cloud.azure.storage.storagewrite

• cloud.azure.storage.storagedelete

• cloud.azure.traffic_manager.probe_health_status

Bug Fix

• The timezone of pendulum.now() is explicitly set to UTC now

• Corrected typo in cloud.azure.intune.operation rules

Improvements

• Updated SDK from 1.12.2 to 1.15.0

Improvements

• Add extend_tag feature to the dynamic tagging system.

• Parameter events_use_autocategory_value is deprecated. The collector always uses autocategory.

• Improved documentation:

  • Updated to last changes, including extend_tag

  • In sync with external documents

  • Shorter user guide, auto categorization in different file

  • All the examples are both in Yaml and Json

Feature

• Added Intune Service

Improvements

• Updated DCDSK from 1.11.1 to 1.12.2

• Fixed high vulnerability in Docker Image

• Upgrade DevoSDK dependency to version v5.4.0

• Fixed error in persistence system

• Applied changes to make DCSDK compatible with MacOS

• Added new sender for relay in house + TLS

• Added persistence functionality for gzip sending buffer

• Added Automatic activation of gzip sending

• Improved behaviour when persistence fails

• Upgraded DevoSDK dependency

• Fixed console log encoding

• Restructured python classes

• Improved behaviour with non-utf8 characters

• Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB)

• New persistence format/structure (compression in some cases)

• Removed dmesg execution (It was invalid for docker execution)

Improvements

• Complete reimplementation of the collector, refactoring all the services

Improvements

• Updated DCSDK from 1.10.3 to 1.11.0

  • Resolution the UTF16 issues.

  • Fixed some bug related to the development.

Improvements

• Update DCSDK from 1.9.2 to 1.10.3:

• Updated DevoSDK to v5.1.9

• Fixed some bug related to development on MacOS

• Added an extra validation and fix when the DCSDK receives a wrong timestamp format

• Added an optional config property for use the Syslog timestamp format in a strict way

Bug fixes

• A bug related to UTF-16 causing the collector to stop sending events

Bug fixes

• Azure metrics were using the incorrect timestamp format which caused logs to go to unknown

Improvements

• Update DCSDK from 1.8.0 to 1.9.2:

  • Upgrade internal dependencies

  • Store lookup instances into DevoSender to avoid creation of new instances for the same lookup

  • Ensure service_config is a dict into templates

  • Ensure special characters are properly sent to the platform

  • Changed log level to some messages from info to debug

  • Changed some wrong log messages

  • Upgraded some internal dependencies

  • Changed queue passed to setup instance constructor

• Update internal Azure libraries

Bug fixes

• Enhancement for event category calculation

Improvements

• Update DCSDK from 1.3.0 to 1.8.0:

  • Added log traces for knowing the execution environment status (debug mode)

  • Fixes in the current puller template version

  • The Docker container exits with the proper error code

  • New controlled stopping condition when any input thread fatally fails

  • Improved log trace details when runtime exceptions happen

  • Refactored source code structure

  • New "templates" functionality

  • Functionality for detecting some system signals for starting the controlled stopping

  • Input objects sends again the internal messages to devo.collectors.out table

  • Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo

  • Refactored source code structure

  • Changed way of executing the controlled stopping

  • Minimized probabilities of suffering a DevoSDK bug related to "sender" to be null

  • Ability to validate collector setup and exit without pulling any data

  • Ability to store in the persistence the messages that couldn't be sent after the collector stopped

  • Ability to send messages from the persistence when the collector starts and before the puller begins working

  • Ensure special characters are properly sent to the platform

  • Added a lock to enhance sender object

  • Added new class attrs to the setstate and getstate queue methods

  • Fix sending attribute value to the setstate and getstate queue methods

  • Added log traces when queues are full and have to wait

  • Added log traces of queues time waiting every minute in debug mode

  • Added method to calculate queue size in bytes

  • Block incoming events in queues when there are no space left

  • Send telemetry events to Devo platform

  • Changed

  • Upgraded internal Python dependency Redis to v4.5.4

  • Upgraded internal Python dependency DevoSDK to v5.1.3

  • Fixed obfuscation not working when messages are sent from templates

Bug fixes

• Updated Azure libraries for Python are updated to share common cloud patterns.

• Change in the authentication mechanism:

  • Previous version: Used ServicePrincipalCredentials in azure.common to authenticate to Azure.

  • New version: Uses the azure.identity library to provide unified authentication for all Azure SDKs.

Bug fixing

• Accept a batch of events that come as an array.

• Filter out non-VM-related events in the SourceSystem branch.

Improvements

• Upgraded underlay IFC SDK v1.3.0 to v1.4.0.

• Updated the underlying DevoSDK package to v3.6.4 and dependencies, this upgrade increases the resilience of the collector when the connection with Devo or the Syslog server is lost. The collector is able to reconnect in some scenarios without running the self-kill feature.

• Support for stopping the collector when a GRACEFULL_SHUTDOWN system signal is received.

• Re-enabled the logging to devo.collector.out for Input threads.

• Improved self-kill functionality behavior.

• Added more details in log traces.

• Added log traces for knowing system memory usage.

Improvements

New events types are accepted for the service vm_events autocategorizer.

• cloud.azure.vm.securityevent:

  • Type: Event

  • EventID: all

  • EventLog: Security

• cloud.azure.vm.applicationevent:

  • Type: Event

  • EventID: all

  • EventLog: Application

• cloud.azure.vm.systemevent:

  • Type: Event

  • EventID: all

  • EventLog: System

Bug fixes

A configuration bug has been fixed to enable the autocategorization of the following events

• RiskyUsers

• AzurePolicyEvaluationDetails