Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 19 Next »

Overview

Wiz is a cloud infrastructure security tool that provides organizations with an in-depth contextual risk assessment. Wiz’s agentless solution builds inventory, and scans for varied risk factors such as vulnerabilities, excessive permissions, malware, exposed secrets, practical exposure, and more, and prioritizes the alerts for the security teams based on the likelihood to be exploited and potential business impact.

The Devo Wiz collector allows customers to retrieve Wiz cloud security issues into Devo to query, correlate, analyze, and visualize to enable Enterprise IT and Cybersecurity teams to take the most impactful decisions at the petabyte scale. The collector processes the Wiz API responses and sends them to the Devo platform, which then categorizes all data received on tables along rows and columns in your Devo domain.

Data sources

Data source

Description

API Endpoint

Collector service name

Devo table

Available from release

Issues

An issue in wiz is a vulnerability that is detected in the cloud infrastructure

/graphql

issues

cspm.wiz.issues.default

v1.0.0

Vulnerability

Vulnerabilities are weaknesses in computer systems that can be exploited by malicious attackers. Whether they are caused by bugs or design flaws, vulnerabilities can allow attackers to execute code in an environment or elevate privileges.

/graphql

vulnerabilities

cspm.wiz.vulnerabilities.default

v1.5.0

Audit Logs

The Audit Log records key events in Wiz, such as login, logout, and user update. The Audit Log is primarily used to investigate potentially suspicious activity or diagnose and troubleshoot errors.

/graphql

auditLogs

cspm.wiz.audit.default

v1.5.0

Cloud Configuration Findings

This returns the problems with configurations and the remediation solutions for the same.

/graphql

cloudConfiguration

cspm.wiz.cloud_configuration.default

v1.5.0

Devo collector features

Feature

Details

Allow parallel downloading (multipod)

Not allowed

Running environments

Collector Server, On Premise

Populated Devo events

Table

Flattening preprocessing

Yes

Flattening preprocessing

In order to improve the data exploitation and enrichment, this collector applies some flattening actions to the collected data before delivering it to Devo:

Data source

Collector service

Optional

Flattening details

Issues

issues

No

  • The control key content is transferred to the first JSON level with the prefix control_.

  • The  entity  key content is transferred to the first JSON level with the prefix entity_.

  • The entitySnapshot key content is transferred to the first JSON level with the prefix entitySnapshot_.

Vulnerabilities

vulnerabilities

Yes

  • The layer key content is transferred to the first json level with the prefix layer_.

  • The vulnerable_asset key content is transferred to the first json level with the prefix asset_.

Audit Logs

auditLogs

Yes

  • The action_parameters key content is transferred to the first json level with the prefix action_.

Cloud Configuration Findings

cloudConfiguration

Yes

  • The resource key content is transferred to the first json level with the prefix resource_.

How to enable the collection in the vendor

Minimal requirements to follow this guide

In order to retrieve the data, the following details will be required from your Wiz instance.

Instance domain

Wiz domain of your cloud instance where the collector will make the requests.

Client ID

Wiz user ID.

Client secret

Wiz user passwords.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.

This minimum configuration refers exclusively to the specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

override_api_base_url

By default, the base URLis https://api.us1.app.wiz.io. This parameter allows you to customize the base URL and is mandatory when the customer ULR is different than the given default value.

client_id

User Client ID to authenticate to the service.

client_secret

User Secret Key to authenticate to the service.

Accepted authentication methods

The following are the accepted authentication methods for this collector.

Authentication method

Client ID

Client secret

Basic authentication

REQUIRED

REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector service details

Issue Service

 Devo categorization and destination

All events of this service are ingested into the table cspm.wiz.issues.default

 Used GraphQL command

Issue service is based on the following GraphQL command:

query IssuesTable(
  $filterBy: IssueFilters
  $first: Int
  $after: String
  $orderBy: IssueOrder
) {
  issues(
    filterBy: $filterBy
    first: $first
    after: $after
    orderBy: $orderBy
  ) {
    nodes {
      ...IssueDetails
    }
    pageInfo {
      hasNextPage
      endCursor
    }
    totalCount
    informationalSeverityCount
    lowSeverityCount
    mediumSeverityCount
    highSeverityCount
    criticalSeverityCount
    uniqueEntityCount
  }
}

fragment IssueDetails on Issue {
  id
  control {
    id
    name
    query
    securitySubCategories {
      id
      title
      category {
        id
        name
        framework {
          id
          name
        }
      }
    }
  }
  createdAt
  updatedAt
  projects {
    id
    name
    slug
    businessUnit
    riskProfile {
      businessImpact
    }
  }
  status
  severity
  entity {
    id
    name
    type
  }
  entitySnapshot {
    id
    type
    nativeType
    name
    subscriptionId
    subscriptionExternalId
    subscriptionName
    resourceGroupId
    resourceGroupExternalId
    region
    cloudPlatform
    cloudProviderURL
    providerId
    status
    tags
    subscriptionTags
  }
  note
  serviceTicket {
    externalId
    name
    url
  }
  serviceTickets {
    externalId
    name
    url
    action {
      id
      type
    }
  }
}
 Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> successfully generated new access token
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Terminated
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Started.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
WARNING InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Terminated
2INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Started
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 5 issues delivered
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Updating deduplication buffers content
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Terminated
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

Vulnerability Service

 Devo categorization and destination

All events of this service are ingested into the table cspm.wiz.vulnerabilities.default

 Used GraphQL command

Issue service is based on the following GraphQL command:

query VulnerabilityFindingsPage(
  $filterBy: VulnerabilityFindingFilters
  $first: Int
  $after: String
  $orderBy: VulnerabilityFindingOrder
) {
  vulnerabilityFindings(
    filterBy: $filterBy
    first: $first
    after: $after
    orderBy: $orderBy
  ) {
    nodes {
      id
      portalUrl
      name
      CVEDescription
      CVSSSeverity
      score
      exploitabilityScore
      impactScore
      dataSourceName
      hasExploit
      hasCisaKevExploit
      status
      vendorSeverity
      firstDetectedAt
      lastDetectedAt
      resolvedAt
      description
      remediation
      detailedName
      version
      fixedVersion
      detectionMethod
      link
      locationPath
      resolutionReason
      epssSeverity
      epssPercentile
      epssProbability
      validatedInRuntime
      layerMetadata {
        id
        details
        isBaseLayer
      }
      projects {
        id
        name
        slug
        businessUnit
        riskProfile {
          businessImpact
        }
      }
      ignoreRules {
        id
        name
        enabled
        expiredAt
      }
      vulnerableAsset {
        ... on VulnerableAssetBase {
          id
          type
          name
          region
          providerUniqueId
          cloudProviderURL
          cloudPlatform
          status
          subscriptionName
          subscriptionExternalId
          subscriptionId
          tags
          hasLimitedInternetExposure
          hasWideInternetExposure
          isAccessibleFromVPN
          isAccessibleFromOtherVnets
          isAccessibleFromOtherSubscriptions
        }
        ... on VulnerableAssetVirtualMachine {
          operatingSystem
          ipAddresses
        }
        ... on VulnerableAssetServerless {
          runtime
        }
        ... on VulnerableAssetContainerImage {
          imageId
        }
        ... on VulnerableAssetContainer {
          ImageExternalId
          VmExternalId
          ServerlessContainer
          PodNamespace
          PodName
          NodeName
        }
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}
 Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> This is the first run of the collector. Generating the access token.
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> successfully generated new access token
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> Puller Setup Terminated
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-02-15T06:48:00.286    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> PrePull Started.
2024-02-15T06:48:00.286    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> User has specified 2023-11-11 20:10:02 as the datetime. Historical polling will consider this datetime for creating the default values.
2024-02-15T06:48:00.286 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Historic datetime in the persistence object and in the configuration are different. Updating the value in state with the user specified datetime.
2024-02-15T06:48:00.286 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Next poll will consider 2023-11-11 20:10:02 to now as the date range.
2024-02-15T06:48:00.287 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-02-15T06:48:00.287    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> PrePull Terminated
2024-02-15T06:48:00.287    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Starting data collection every 60 seconds
2024-02-15T06:48:00.287    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Pull Started
2024-02-15T06:48:00.288    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Fetching vulnerabilities from 2023-11-11T20:10:02Z
2024-02-15T06:48:00.288    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Requesting Wiz API for vulnerabilities
2024-02-15T06:48:03.155    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> successfully retried vulnerabilities from Wiz
2024-02-15T06:48:03.240    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:03.241    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:03.250    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:03.255    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:03.485    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> 500 issues delivered
2024-02-15T06:48:03.507    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 13, 14, 38, 14, 622387), 'buffer_ids_with_duplication_risk': ['f7723a4c-3108-5149-8c5b-52582c2a6474']}.
2024-02-15T06:48:03.507    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.286006) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:03.507    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Requesting Wiz API for vulnerabilities
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
2024-02-15T06:48:06.423    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> successfully retried vulnerabilities from Wiz
2024-02-15T06:48:06.454    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:06.457    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:06.459    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:06.509    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> 500 issues delivered
2024-02-15T06:48:06.510    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 17, 21, 54, 52, 964304), 'buffer_ids_with_duplication_risk': ['23d093be-97a8-59f9-a95d-36a841943235']}.
2024-02-15T06:48:06.510    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.286006) so far: Number of requests made: 2; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000.
2024-02-15T06:48:06.510    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Requesting Wiz API for vulnerabilities
2024-02-15T06:57:59.335    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> successfully retried vulnerabilities from Wiz
2024-02-15T06:57:59.336    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Removing the duplicate issues if present
2024-02-15T06:57:59.337    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:57:59.337    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Delivering issues to the SDK
2024-02-15T06:57:59.337    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> 0 issues delivered
2024-02-15T06:57:59.338    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Updating deduplication buffers content
2024-02-15T06:57:59.338    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707979680.286006):Number of requests made: 196; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 97500; Average of events per second: 162.758.
2024-02-15T06:57:59.338    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Pull Terminated
2024-02-15T07:04:28.645    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Data collection completed. Elapsed time: 0.626 seconds. Waiting for 59.374 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-02-15T07:03:39.203    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707980279.338879) so far: Number of requests made: 110; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 55000.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

AuditLogs Service

 Devo categorization and destination

All events of this service are ingested into the table cspm.wiz.audit.default

 Used GraphQL command

Issue service is based on the following GraphQL command:

query AuditLogTable(
  $first: Int
  $after: String
  $filterBy: AuditLogEntryFilters
) {
  auditLogEntries(first: $first, after: $after, filterBy: $filterBy) {
    nodes {
      id
      action
      requestId
      status
      timestamp
      actionParameters
      userAgent
      sourceIP
      serviceAccount {
        id
        name
      }
      user {
        id
        name
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}
 Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2024-02-15T06:47:57.287    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Puller Setup Started
2024-02-15T06:47:57.287    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> This is the first run of the collector. Generating the access token.
2024-02-15T06:47:57.288    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Getting the auth token url based on provided api_base_url
2024-02-15T06:47:57.288    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Getting the auth token url based on provided api_base_url
2024-02-15T06:47:59.494    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Puller Setup Terminated
2024-02-15T06:47:59.494    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-02-15T06:48:00.295    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> PrePull Started.
2024-02-15T06:48:00.295    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> User has specified 2023-11-11 20:10:02 as the datetime. Historical polling will consider this datetime for creating the default values.
2024-02-15T06:48:00.295 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Historic datetime in the persistence object and in the configuration are different. Updating the value in state with the user specified datetime.
2024-02-15T06:48:00.295 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Next poll will consider 2023-11-11 20:10:02 to now as the date range.
2024-02-15T06:48:00.296 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> PrePull Terminated
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Starting data collection every 60 seconds
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Pull Started
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Fetching auditLogs from 2023-11-11T20:10:02Z
2024-02-15T06:48:00.297    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Requesting Wiz API for auditLogs
2024-02-15T06:48:01.886    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> successfully retried auditLogs from Wiz
2024-02-15T06:48:01.918    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:01.920    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:02.017    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> 500 issues delivered
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2024, 2, 15, 6, 47, 59, 126416), 'buffer_ids_with_duplication_risk': []}.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.295365) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Requesting Wiz API for auditLogs
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
2024-02-15T06:48:01.886    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> successfully retried auditLogs from Wiz
2024-02-15T06:48:01.918    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:01.920    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:02.017    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> 500 issues delivered
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2024, 2, 15, 6, 47, 59, 126416), 'buffer_ids_with_duplication_risk': []}.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.295365) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Requesting Wiz API for auditLogs
2024-02-15T06:48:32.320    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> successfully retried auditLogs from Wiz
2024-02-15T06:48:32.326    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:32.328    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:32.328    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:32.343    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> 343 issues delivered
2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 16, 1, 16, 19, 908609), 'buffer_ids_with_duplication_risk': []}.
2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Updating deduplication buffers content
2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707979680.295365):Number of requests made: 21; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 10343; Average of events per second: 322.735.
2024-02-15T06:48:32.345    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Pull Terminated
2024-02-15T06:48:32.345    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Data collection completed. Elapsed time: 32.050 seconds. Waiting for 27.950 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707979680.295365):Number of requests made: 21; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 10343; Average of events per second: 322.735.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

CloudConfiguration Service

 Devo categorization and destination

All events of this service are ingested into the table cspm.wiz.cloud_configuration.default

 Used GraphQL command

Issue service is based on the following GraphQL command:

query CloudConfigurationFindingsPage(
  $filterBy: ConfigurationFindingFilters
  $first: Int
  $after: String
  $orderBy: ConfigurationFindingOrder
) {
  configurationFindings(
    filterBy: $filterBy
    first: $first
    after: $after
    orderBy: $orderBy
  ) {
    nodes {
      id
      targetExternalId
      targetObjectProviderUniqueId
      firstSeenAt
      severity
      result
      status
      remediation
      resource {
        id
        providerId
        name
        nativeType
        type
        region
        subscription {
          id
          name
          externalId
          cloudProvider
        }
        projects {
          id
          name
          riskProfile {
            businessImpact
          }
        }
        tags {
          key
          value
        }
      }
      rule {
        id
        graphId
        name
        description
        remediationInstructions
        functionAsControl
      }
      securitySubCategories {
        id
        title
        category {
          id
          name
          framework {
            id
            name
          }
        }
      }
      ignoreRules {
        id
        name
        enabled
        expiredAt
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}
 Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2024-02-15T06:47:57.290    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Puller Setup Started
2024-02-15T06:47:57.292    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> This is the first run of the collector. Generating the access token.
2024-02-15T06:47:57.292    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Getting the auth token url based on provided api_base_url
2024-02-15T06:47:57.292    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Using default Authentication Domain auth.wiz.io for fetching Access Token
2024-02-15T06:47:59.891    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Puller Setup Terminated
2024-02-15T06:47:59.891    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-02-15T06:48:00.290    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> PrePull Started.
2024-02-15T06:48:00.291    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> User has specified 2023-11-11 20:10:02 as the datetime. Historical polling will consider this datetime for creating the default values.
2024-02-15T06:48:00.291 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Historic datetime in the persistence object and in the configuration are different. Updating the value in state with the user specified datetime.
2024-02-15T06:48:00.291 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Next poll will consider 2023-11-11 20:10:02 to now as the date range.
2024-02-15T06:48:00.291 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> PrePull Terminated
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Starting data collection every 60 seconds
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Pull Started
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Fetching cloudConfiguration from 2023-11-11T20:10:02Z
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
2024-02-15T06:48:29.044    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> successfully retried cloudConfiguration from Wiz
2024-02-15T06:48:29.127    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:29.128    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:29.128    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:29.129    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:29.238    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> 500 issues delivered
2024-02-15T06:48:29.239    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 23, 6, 46, 39, 979510), 'buffer_ids_with_duplication_risk': ['498e0032-f76f-58d8-935b-dc20546c4d77']}.
2024-02-15T06:48:29.239    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.290731) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:29.239    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
2024-02-15T06:48:40.162    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> successfully retried cloudConfiguration from Wiz
2024-02-15T06:48:40.207    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:40.216    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:40.217    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:40.297    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> 500 issues delivered
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 24, 3, 29, 59, 755884), 'buffer_ids_with_duplication_risk': ['0e446865-0e54-567d-a3ab-f5bfedecefbc']}.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.290731) so far: Number of requests made: 2; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
2024-02-15T06:48:40.162    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> successfully retried cloudConfiguration from Wiz
2024-02-15T06:48:40.207    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:40.216    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:40.217    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:40.297    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> 500 issues delivered
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 24, 3, 29, 59, 755884), 'buffer_ids_with_duplication_risk': ['0e446865-0e54-567d-a3ab-f5bfedecefbc']}.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.290731) so far: Number of requests made: 2; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
2024-02-15T07:01:14.752    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707980301.689743):Number of requests made: 26; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 12779; Average of events per second: 73.841.
2024-02-15T07:01:14.752    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Pull Terminated

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-02-15T07:01:14.752    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707980301.689743):Number of requests made: 26; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 12779; Average of events per second: 73.841.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

 Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

  1. Edit the configuration file.

  2. Change the value of the historical_date_utc parameter to a different one.

  3. Save the changes.

  4. Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

 Troubleshooting

Error type

Error ID

Error message

Cause

Solution

InitVariablesError

1

Devo tag is the required field for sending events to Devo. Specify it in collector definitions

This error is raised when devo_tag property is not found in collector_definitions.yaml.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

2

Required setting. devo_tag is not of expected type: str

This error is raised when devo_tag is defined in collector_definitions.yaml but the format is not str.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

3

Optional setting, override_devo_tag not of expected type: str

This error is raised when optional value override_devo_tag added in config.json is not of type str.

Edit the value of override_devo_tag in config.json so it is of type str. Or leave it empty so it takes the default value.

InitVariablesError

4

GraphQL query is the required field for querying issues from Wiz. Specify it in collector definitions

This error is raised when graphql_query is not found in collector_definitions.yaml.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

5

Required setting. graphql_query is not of expected type: str

This error is raised when graphql_query defined in collector_definitions.yaml is not of type str.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

6

user_agent is the required field for passing in headers of Wiz API calls. Specify it in collector definitions

This error is raised when user_agent is not found in collector_definitions.yaml.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

7

Required setting. user_agent is not of expected type: str

This error is raised when user_agent defined in collector_definitions.yaml is not of type str.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

8

Optional setting, flatten_data not of expected type: bool

This error is raised when the optional value flatten_data defined in config.json is not of type bool.

Edit the value of flatten_data in config.json so it is of type bool. You can also remove override_flatten_data parameter from config.json so it takes the default value.

InitVariablesError

9

Optional setting, requests_per_second not of expected type: int

This error is raised when the optional value requests_per_second defined in config.json is not of type int.

Edit the value of requests_per_second in config.json so it is of type int. Or leave it empty so it takes the default value.

InitVariablesError

10

Required setting. requested_page_size_in_items is not of expected type: int

This error is raised when requested_page_size_in_items defined in collector_definitions.yaml is not of type int.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

11

access_token_timeout is the required field for checking if the token is expired. Specify it in collector definitions

This error is raised when access_token_timeout is not found in collector_definitions.yaml.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

12

Required setting. access_token_timeout is not of expected type: int

This error is raised when access_token_timeout defined in collector_definitions.yaml is not of type int.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

13

default_historic_days is the required field in case historic_date_utc is not specified. Specify it in collector definitions

This error is raised when default_historic_days is not found in collector_definitions.yaml.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

14

Required setting. default_historic_days is not of expected type: int

This error is raised when default_historic_days defined in collector_definitions.yaml is not of type int.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

15

api_url_regex is the required field for validating the base url. Specify it in collector definitions

This error is raised when api_url_regex is not found in collector_definitions.yaml.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

16

Required setting. api_url_regex is not of expected type: str

This error is raised when api_url_regex defined in collector_definitions.yaml is not of type str.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

17

historic_date_time_format is the required field for validating datetime format. Specify it in collector definitions

This error is raised when api_url_regex is not found in collector_definitions.yaml.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

18

Required setting. historic_date_time_format is not of expected type: str

This error is raised when api_url_regex defined in collector_definitions.yaml is not of type str.

This is an internal issue. Contact with Devo Support team.

InitVariablesError

19

api_base_url not of expected type: str

This error could be raised for two reasons:

  1. api_base_url defined in collector_definitions.yaml is not of type str.

  2. override_api_base_url defined in config.json is not of type str.

Solutions for both cases, respectively:

  1. This is an internal issue. Contact with Devo Support team.

  2. Edit the value of override_api_base_url in config.json so it is of type str. Or remove the parameter definition, so it takes the default value.

InitVariablesError

20

api_base_url must match regex: <regex>

This error could be raised for two reasons:

  1. api_base_url defined in collector_definitions.yaml does not match the required regex.

  2. override_api_base_url defined in config.json does not match the required regex.

Solutions for both cases, respectively:

  1. This is an internal issue. Contact with Devo Support team.

  2. Edit the value of override_api_base_url in config.json so it matches the indicated regex. Or remove the parameter definition so it takes the default value. If default value was used it is an internal issue, so contact with Devo Support team.

InitVariablesError

21

Required setting, credentials not found in user configuration

This error is raised when the required property credentials is not found in config.json.

Add credentials dictionary in config.json, including client_id and client_secret fields.

InitVariablesError

22

Required setting, credentials not of expected type: dict

This error is raised when credentials is defined in config.json but the format is not dict.

Edit the value of credentials in config.json so it is of type dict.

InitVariablesError

23

Required setting, client_id not found in user configuration

This error is raised when the required property client_id is not found in config.json, into credentials dictionary.

Add client_id property in config.json, into credentials dictionary.

InitVariablesError

24

Required setting, client_id not of expected type: str

This error is raised when client_id is defined in config.json but the format is not str.

Edit the value of client_id in config.json, into credentials dictionary, so it is of type str.

InitVariablesError

25

Required setting, client_secret not found in user configuration

This error is raised when the required property client_secret is not found in config.json, into credentials dictionary.

Add client_secret property in config.json, into credentials dictionary.

InitVariablesError

26

Required setting, client_secret not of expected type: str

This error is raised when client_secret is defined in config.json but the format is not str.

Edit the value of client_secret in config.json, into credentials dictionary, so it is of type str.

InitVariablesError

27

Required setting, historic_date_utc not of expected type: str

This error is raised when the optional value historic_date_utc defined in config.json is not of type str.

Edit the value of historic_date_utc in config.json so it is of type str. Or leave it empty so the collector starts pulling data N days ago at the current time.

InitVariablesError

28

Time format for historic date must be <time_format>. e.g. 2022-02-15T14:32:33.043Z

This error is raised when the optional value historic_date_utc defined in config.json does not match the indicated format.

Make the value of historic_date_utc in config.json match the indicated format. Or leave it empty so the collector starts pulling data N days ago at the current time.

InitVariablesError

29

historic datetime cannot be greater than the present UTC time

This error is raised when the optional value historic_date_utc defined in config.json is higher than the current time. Time is indicated in UTC time.

Make the value of historic_date_utc in config.json be a past value in UTC time. Or leave it empty so the collector starts pulling data N days ago at the current time.

SetupError

100

Error occurred while requesting access token from the Wiz server. Error message: <error_message>

This error is raised when a generic error occurs during the request to get the token to authenticate the collector in the API.

This is an internal issue. Contact with Devo Support team.

SetupError

101

The credentials provided in the config file are incorrect. Please provide the correct credentials.

Status code: 401

Error type: <error_type>

Error message: <error_message>

This error is raised when the credentials provided in config.json are not valid. (401 Unauthorized error)

Add the correct client_id and client_secret fields in config.json, at credentials property.

SetupError

102

The credentials provided in the config file does not have necessary permissions to create access token.

Status code: 403

Error type: <error_type>

Error message: <error_message>

This error is raised when the credentials provided in config.json are valid, but the credentials used to access the API have no permissions to create a token. (403 Forbidden error)

Add some credentials in config.json with enough privileges to create a token from the API.

SetupError

103

The requested URL <URL> is not found. The URL may have been deprecated.

Status code: 404

Error message: <error_message>

This error is raised when the credentials provided in config.json are valid, but the authentication endpoint that is being requested to get a token is not found.

This is an internal issue. Contact with Devo Support team.

SetupError

104

Unexpected error occurred while getting access token from the Wiz server

Status code: <status_code>

Error message: <error_message>

This error is raised when the credentials provided in config.json are valid, but an unexpected response has been returned from the API.

This is an internal issue. Contact with Devo Support team.

SetupError

105

The credentials does not have valid permissions to fetch issues from the Wiz server

This error is raised when the credentials provided in config.json are valid, but the credentials used have no permissions to access the API endpoint to get issues.

Check that the provided credentials have enough permission to retrieve issues from the API.

SetupError

106

Failed to check if the provided credentials have valid permissions.

Error message: <error_message>

This error is raised when the credentials provided in config.json are valid, but there was an error in the process of checking if the provided credentials have permission to get issues.

This is an internal issue. Contact with Devo Support team.

PullError

300

Error occurred while requesting issues from the Wiz server. Error message: <error_message>

This error is raised when a generic error occurs during the request to get issues from the API.

This is an internal issue. Contact with Devo Support team.

PullError

301

The token used to make this request is not valid anymore.

Status code: 401

Error message: <error_message>

This error is raised when the token being used to make requests to the API is not valid anymore.

Check if the credentials need to be renewed. Add some valid credentials in config.json.

If credentials are still valid, contact with Devo Support team.

PullError

302

The access token does not have necessary permissions to fetch issues from Wiz.

Status code: 403

Error message: <error_message>

This error is raised when the token being used to make requests to the API is valid, but it lost permissions to get issues from API.

Check what happened with credential permissions. Add some credentials in config.json with permissions to get issues.

If credentials are valide and still have permissions, contact with Devo Support team.

PullError

303

The requested URL <URL> is not found. The URL may have been depreciated

Status code: 404

Error message: <error_message>

This error is raised when the token being used to make requests to the API is valid, but cannot find the endpoints to get issues.

This is an internal issue. Contact with Devo Support team.

PullError

304

The server has returned <status_code> status code. The server may not be available for fetching issues. Try after sometime. Error message from server: <error_message>

This error is raised when the token being used to make requests to the API is valid, but there has been an error on Wiz's API.

The error is on Wiz’s side. Wiz can be contacted for more info.

It should work again when the incident at Wiz is solved.

PullError

305

Unexpected error occurred while getting issues from the Wiz server

Status code: <status_code>

Error message: <error_message>

This error is raised when the token being used to make requests to the API is valid, but there has been an unexpected return from the API.

This is an internal issue. Contact with Devo Support team.

PullError

306

After <retry_count> retries still getting the too many requests error.

This error is raised when the token being used to make requests to the API is valid, but we are constantly receiving a 429 error response (too many requests)

Check throttle limitations on Wiz API and change the value of request_period_in_seconds and xxxxxxxx from config.json to adapt to it.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

 Verify collector operations

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 1000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-eu.devo.io', 'port': 443, ...}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
INFO OutputProcess::MainThread -> Process started
INFO InputProcess::MainThread -> Process Started
INFO InputProcess::MainThread -> InitVariables Started
INFO InputProcess::MainThread -> Validating variables in collector definitions Started
INFO InputProcess::MainThread -> Flatten data is not provided in the config.yaml. Considering the flatten data from collector definitions
INFO InputProcess::MainThread -> Validating collector Variables is terminated
INFO InputProcess::MainThread -> Initialization of api_base_url has started.
INFO InputProcess::MainThread -> api_base_url has been initialized
INFO InputProcess::MainThread -> Initialization of credentials has started.
INFO InputProcess::MainThread -> credentials have been initialized.
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO InputProcess::MainThread -> InitVariables Terminated
INFO InputProcess::MainThread -> InputThread(wiz_data_puller,111) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> ServiceThread(wiz_data_puller,111,issues,predefined) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Starting thread
INFO InputProcess::MainThread -> WizDataPuller(wiz_data_puller,111,issues,predefined) - Starting thread
WARNING InputProcess::WizDataPuller(wiz_data_puller,111,issues,predefined) -> Waiting until setup will be executed
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> This is the first run of collector. Generating the access token
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Getting the auth token url based on provided api_base_url
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Using default Authentication Domain auth.wiz.io for fetching Access Token
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
INFO InputProcess::MainThread -> [GC] global: 36.7% -> 36.7%, process: RSS(26.93MiB -> 27.97MiB), VMS(334.43MiB -> 334.67MiB)
INFO OutputProcess::MainThread -> [GC] global: 36.7% -> 36.3%, process: RSS(26.68MiB -> 28.61MiB), VMS(910.71MiB -> 910.71MiB)
INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-eu.devo.io:443", ...}

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)

By default, these information traces will be displayed every 10 minutes.

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services

Description

internal_senders

In charge of delivering internal metrics to Devo such as logging traces or metrics.

standard_senders

In charge of delivering pulled events to Devo.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

  • 44 events were sent to Devo since the collector started.

  • The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.

  • 21 events where sent to Devo between the last UTC checkpoint and now.

  • Those 21 events required 0.007 seconds to be delivered.

By default these traces will be shown every 10 minutes.

 Check memory usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

  • The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

  • The global pressure of the available memory is displayed in the global value.

  • All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)

Differences between RSS and VMS memory usage:

  • RSS is the Resident Set Size, which is the actual physical memory the process is using

  • VMS is the Virtual Memory Size which is the virtual memory that process is using

 Enable/disable the logging debug mode

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

  • To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.

  • To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Change log

Release

Released on

Release type

Details

Recommendations

v1.5.0

FEATUREIMPROVEMENTS

New Features

  • Added following three new services:

    • Vulnerabilities

    • Audit Logs

    • Cloud Configuration Findings

Improvements

  • Upgraded DCSDK from 1.10.2 to 1.10.3 .

Upgrade

v1.4.0

FEATUREIMPROVEMENTS

New Features

  • Added extra filters for events:

    • type: Filter by Issue type. You can specify multiple values in an array.

      • Possible values: ["TOXIC_COMBINATION", "THREAT_DETECTION", "CLOUD_CONFIGURATION"]

Improvements

  • Upgraded DCSDK from 1.9.2 to 1.10.2

    • Added input metrics

    • Modified ouutput metrics

    • Updated DevoSDK to version 5.1.6

    • Standardized exception messages for traceability

    • Added more detail in queue statistics

    • Updated PythonSDK to version 5.0.7

    • Introduced pyproject.toml

    • Added requirements.dev.txt

    • Fixed error in pyproject.toml related to project scripts endpoint

Recommended

v1.3.0

BUG FIXIMPROVEMENTS

Improvements:

  • Upgraded DCSDK from 1.9.1 to 1.9.2

    • upgraded dependencies

Bug Fix:

  • Remove actions from service tables

Recommended

v1.2.0

IMPROVEMENTS

Improvements:

  • Upgraded DCSDK from 1.3.0 to 1.9.1

    • Store lookup instances into DevoSender to avoid creation of new instances for the same lookup

    • Ensure service_config is a dict into templates

    • Ensure special characters are properly sent to the platform

    • Changed log level to some messages from info to debug

    • Changed some wrong log messages

    • Upgraded some internal dependencies

    • Changed queue passed to setup instance constructor

    • Added log traces for knowing the execution environment status (debug mode)

    • Fixes in the current puller template version

    • Improved log trace details when runtime exceptions happen

    • Refactored source code structure

    • New “templates” functionality

    • Functionality for detecting some system signals for starting the controlled stopping

    • Input objects sends again the internal messages to devo.collectors.out table

    • Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo

    • Refactored source code structure

    • Changed way of executing the controlled stopping

    • Minimized probabilities of suffering a DevoSDK bug related to “sender” to be null

    • Ability to validate collector setup and exit without pulling any data

    • Ability to store in the persistence the messages that couldn’t be sent after the collector stopped

    • Ability to send messages from the persistence when the collector starts and before the puller begins working

    • Ensure special characters are properly sent to the platform

    • Added a lock to enhance sender object

    • Added new class attrs to the __setstate__ and __getstate__ queue methods

    • Fix sending attribute value to the __setstate__ and __getstate__ queue methods

    • Added log traces when queues are full and have to wait

    • Added log traces of queues time waiting every minute in debug mode

    • Added method to calculate queue size in bytes

    • Block incoming events in queues when there are no space left

    • Send telemetry events to Devo platform

    • Upgraded internal Python dependency Redis to v4.5.4

    • Upgraded internal Python dependency DevoSDK to v5.1.3

    • Fixed obfuscation not working when messages are sent from templates

    • New method to figure out if a puller thread is stopping

    • Upgraded internal Python dependency DevoSDK to v5.0.6

    • Improved logging on messages/bytes sent to Devo platform

    • Fixed wrong bytes size calculation for queues

    • New functionality to count bytes sent to Devo Platform (shown in console log)

    • Upgraded internal Python dependency DevoSDK to v5.0.4

    • Fixed bug in persistence management process, related to persistence reset

    • Aligned source code typing to be aligned with Python 3.9.x

    • Inject environment property from user config

    • Obfuscation service can be now configured from user config and module definition

    • Obfuscation service can now obfuscate items inside arrays

    • Ensure special characters are properly sent to the platform

    • The resilience has been improved with a new feature that restart the collector when the Devo connections is lost and it cannot be recovered.

    • When an exception is raised by the Collector Setup, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied.

    • When an exception is raised by the Collector Pull method, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied.

    • When an exception is raised by the Collector pre-pull method, the collector retries after 30 seconds. No maximum retries are applied.

    • Changed log level to some messages from info to debug

    • Changed some wrong log messages

    • Upgraded some internal dependencies

    • Changed queue passed to setup instance constructor

Upgrade

v1.1.1

BUG FIX

Bug fixes:

  • Force using always UTC timezone for all date time operations.

Recommended version

v1.1.0

FEATURE

New features:

  • Wiz’s new authentication via Cognito is now available. Former authentication using Auth0 is also still compatible.

Recommended version

v1.0.0

FEATURE

New features:

  • Wiz issues

Upgrade

  • No labels