Wiz collector

[ 1 Overview ] [ 2 Data sources ] [ 3 Devo collector features ] [ 4 Flattening preprocessing ] [ 5 How to enable the collection in the vendor ] [ 6 Minimum configuration required for basic pulling ] [ 7 Accepted authentication methods ] [ 8 Run the collector ] [ 9 Collector service details ] [ 10 Collector operations ] [ 11 Change log ]

Overview

Wiz is a cloud infrastructure security tool that provides organizations with an in-depth contextual risk assessment. Wiz’s agentless solution builds inventory, and scans for varied risk factors such as vulnerabilities, excessive permissions, malware, exposed secrets, practical exposure, and more, and prioritizes the alerts for the security teams based on the likelihood to be exploited and potential business impact.

The Devo Wiz collector allows customers to retrieve Wiz cloud security issues into Devo to query, correlate, analyze, and visualize to enable Enterprise IT and Cybersecurity teams to take the most impactful decisions at the petabyte scale. The collector processes the Wiz API responses and sends them to the Devo platform, which then categorizes all data received on tables along rows and columns in your Devo domain.

Data sources

Data source	Description	API Endpoint	Collector service name	Devo table	Available from release

Data source	Description	API Endpoint	Collector service name	Devo table	Available from release
Issues	An issue in wiz is a vulnerability that is detected in the cloud infrastructure	`/graphql`	`issues`	`cspm.wiz.issues.default`	`v1.0.0`
Vulnerability	Vulnerabilities are weaknesses in computer systems that can be exploited by malicious attackers. Whether they are caused by bugs or design flaws, vulnerabilities can allow attackers to execute code in an environment or elevate privileges.	`/graphql`	`vulnerabilities`	`cspm.wiz.vulnerabilities.default`	`v1.5.0`
Audit Logs	The Audit Log records key events in Wiz, such as login, logout, and user update. The Audit Log is primarily used to investigate potentially suspicious activity or diagnose and troubleshoot errors.	`/graphql`	`auditLogs`	`cspm.wiz.audit.default`	`v1.5.0`
Cloud Configuration Findings	This returns the problems with configurations and the remediation solutions for the same.	`/graphql`	`cloudConfiguration`	`cspm.wiz.cloud_configuration.default`	`v1.5.0`
Custom Service	This provides an option to add custom graphql query in the config and ingest data.	`/graphql`	`custom_query`	`my.app.wiz.custom_query` (default) User can provide override tag in the config if the parser is deployed for their custom query or if they want a different table in my.app .	`v1.7.0`

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`not allowed`
Running environments	`collector server` `on-premise`
Populated Devo events	`table`
Flattening preprocessing	`yes`

Flattening preprocessing

In order to improve the data exploitation and enrichment, this collector applies some flattening actions to the collected data before delivering it to Devo:

Data source	Collector service	Optional	Flattening details

Data source	Collector service	Optional	Flattening details
Issues	`issues`	`No`	The `control` key content is transferred to the first JSON level with the prefix `control_`. The `entity` key content is transferred to the first JSON level with the prefix `entity_`. The `entitySnapshot` key content is transferred to the first JSON level with the prefix `entitySnapshot_`.
Vulnerabilities	`vulnerabilities`	`Yes`	The `layer` key content is transferred to the first json level with the prefix `layer_`. The `vulnerable_asset` key content is transferred to the first json level with the prefix `asset_`.
Audit Logs	`auditLogs`	`Yes`	The `action_parameters` key content is transferred to the first json level with the prefix `action_`.
Cloud Configuration Findings	`cloudConfiguration`	`Yes`	The `resource` key content is transferred to the first json level with the prefix `resource_`.
Custom Service	`custom_query`	`Yes`	N/A

How to enable the collection in the vendor

Minimal requirements to follow this guide

In order to retrieve the data, the following details will be required from your Wiz instance.

Instance domain	Wiz domain of your cloud instance where the collector will make the requests.
Client ID	Wiz user ID.
Client secret	Wiz user passwords.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.

This minimum configuration refers exclusively to the specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details

Setting	Details
`override_api_base_url`	Set the base URL for the Wiz API. An example is `https://api.us1.app.wiz.io`. This parameter is mandatory.
`client_id`	User Client ID to authenticate to the service.
`client_secret`	User Secret Key to authenticate to the service.

For authentication, Devo will contact auth.app.wiz.io. If customization of the authentication URL is required, contact us.

Accepted authentication methods

The following are the accepted authentication methods for this collector.

Authentication method	Client ID	Client secret

Authentication method	Client ID	Client secret
Basic authentication	status:REQUIRED	status:REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector service details

Issue Service

All events of this service are ingested into the table cspm.wiz.issues.default

Issue service is based on the following GraphQL command:

query IssuesTable(
  $filterBy: IssueFilters
  $first: Int
  $after: String
  $orderBy: IssueOrder
) {
  issues(
    filterBy: $filterBy
    first: $first
    after: $after
    orderBy: $orderBy
  ) {
    nodes {
      ...IssueDetails
    }
    pageInfo {
      hasNextPage
      endCursor
    }
    totalCount
    informationalSeverityCount
    lowSeverityCount
    mediumSeverityCount
    highSeverityCount
    criticalSeverityCount
    uniqueEntityCount
  }
}

fragment IssueDetails on Issue {
  id
  control {
    id
    name
    query
    securitySubCategories {
      id
      title
      category {
        id
        name
        framework {
          id
          name
        }
      }
    }
  }
  createdAt
  updatedAt
  projects {
    id
    name
    slug
    businessUnit
    riskProfile {
      businessImpact
    }
  }
  status
  severity
  entity {
    id
    name
    type
  }
  entitySnapshot {
    id
    type
    nativeType
    name
    subscriptionId
    subscriptionExternalId
    subscriptionName
    resourceGroupId
    resourceGroupExternalId
    region
    cloudPlatform
    cloudProviderURL
    providerId
    status
    tags
    subscriptionTags
  }
  note
  serviceTicket {
    externalId
    name
    url
  }
  serviceTickets {
    externalId
    name
    url
    action {
      id
      type
    }
  }
}

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> successfully generated new access token
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Terminated
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Started.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
WARNING InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Terminated
2INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Started
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 5 issues delivered
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Updating deduplication buffers content
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Terminated
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

Vulnerability Service

All events of this service are ingested into the table cspm.wiz.vulnerabilities.default

Issue service is based on the following GraphQL command:

query IssuesTable(
  $filterBy: IssueFilters
  $first: Int
  $after: String
  $orderBy: IssueOrder
) {
  issues: issuesV2(
    filterBy: $filterBy
    first: $first
    after: $after
    orderBy: $orderBy
  ) {
    nodes {
      ...IssueDetails
    }
    pageInfo {
      hasNextPage
      endCursor
    }
    totalCount
    informationalSeverityCount
    lowSeverityCount
    mediumSeverityCount
    highSeverityCount
    criticalSeverityCount
    uniqueEntityCount
  }
}

fragment IssueDetails on Issue {
  id
  type
  description
  sourceRule {
    __typename
    ... on Control {
      id
      name
      controlDescription: description
      resolutionRecommendation
      securitySubCategories {
        title
        category {
          name
          framework {
            name
          }
        }
      }
    }
    ... on CloudEventRule {
      id
      name
      cloudEventRuleDescription: description
      sourceType
      type
    }
    ... on CloudConfigurationRule {
      id
      name
      cloudConfigurationRuleDescription: description
      remediationInstructions
      serviceType
    }
  }
  control {
    id
    name
    query
    securitySubCategories {
      id
      title
      category {
        id
        name
        framework {
          id
          name
        }
      }
    }
  }
  createdAt
  updatedAt
  projects {
    id
    name
    slug
    businessUnit
    riskProfile {
      businessImpact
    }
  }
  status
  severity
  entity {
    id
    name
    type
  }
  entitySnapshot {
    id
    type
    nativeType
    name
    subscriptionId
    subscriptionExternalId
    subscriptionName
    resourceGroupId
    resourceGroupExternalId
    region
    cloudPlatform
    cloudProviderURL
    providerId
    status
    tags
    subscriptionTags
  }
  note
  serviceTicket {
    externalId
    name
    url
  }
  serviceTickets {
    externalId
    name
    url
  }
}

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> This is the first run of the collector. Generating the access token.
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> successfully generated new access token
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> Puller Setup Terminated
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#00001,vulnerabilities#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-02-15T06:48:00.286    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> PrePull Started.
2024-02-15T06:48:00.286    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> User has specified 2023-11-11 20:10:02 as the datetime. Historical polling will consider this datetime for creating the default values.
2024-02-15T06:48:00.286 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Historic datetime in the persistence object and in the configuration are different. Updating the value in state with the user specified datetime.
2024-02-15T06:48:00.286 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Next poll will consider 2023-11-11 20:10:02 to now as the date range.
2024-02-15T06:48:00.287 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-02-15T06:48:00.287    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> PrePull Terminated
2024-02-15T06:48:00.287    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Starting data collection every 60 seconds
2024-02-15T06:48:00.287    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Pull Started
2024-02-15T06:48:00.288    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Fetching vulnerabilities from 2023-11-11T20:10:02Z
2024-02-15T06:48:00.288    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Requesting Wiz API for vulnerabilities
2024-02-15T06:48:03.155    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> successfully retried vulnerabilities from Wiz
2024-02-15T06:48:03.240    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:03.241    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:03.250    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:03.255    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:03.485    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> 500 issues delivered
2024-02-15T06:48:03.507    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 13, 14, 38, 14, 622387), 'buffer_ids_with_duplication_risk': ['f7723a4c-3108-5149-8c5b-52582c2a6474']}.
2024-02-15T06:48:03.507    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.286006) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:03.507    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Requesting Wiz API for vulnerabilities
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
2024-02-15T06:48:06.423    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> successfully retried vulnerabilities from Wiz
2024-02-15T06:48:06.454    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:06.457    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:06.459    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:06.509    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> 500 issues delivered
2024-02-15T06:48:06.510    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 17, 21, 54, 52, 964304), 'buffer_ids_with_duplication_risk': ['23d093be-97a8-59f9-a95d-36a841943235']}.
2024-02-15T06:48:06.510    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.286006) so far: Number of requests made: 2; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000.
2024-02-15T06:48:06.510    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Requesting Wiz API for vulnerabilities
2024-02-15T06:57:59.335    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> successfully retried vulnerabilities from Wiz
2024-02-15T06:57:59.336    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Removing the duplicate issues if present
2024-02-15T06:57:59.337    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:57:59.337    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Delivering issues to the SDK
2024-02-15T06:57:59.337    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> 0 issues delivered
2024-02-15T06:57:59.338    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Updating deduplication buffers content
2024-02-15T06:57:59.338    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707979680.286006):Number of requests made: 196; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 97500; Average of events per second: 162.758.
2024-02-15T06:57:59.338    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Pull Terminated
2024-02-15T07:04:28.645    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> Data collection completed. Elapsed time: 0.626 seconds. Waiting for 59.374 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-02-15T07:03:39.203    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,vulnerabilities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707980279.338879) so far: Number of requests made: 110; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 55000.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

AuditLogs Service

All events of this service are ingested into the table cspm.wiz.audit.default

Issue service is based on the following GraphQL command:

query AuditLogTable(
  $first: Int
  $after: String
  $filterBy: AuditLogEntryFilters
) {
  auditLogEntries(first: $first, after: $after, filterBy: $filterBy) {
    nodes {
      id
      action
      requestId
      status
      timestamp
      actionParameters
      userAgent
      sourceIP
      serviceAccount {
        id
        name
      }
      user {
        id
        name
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2024-02-15T06:47:57.287    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Puller Setup Started
2024-02-15T06:47:57.287    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> This is the first run of the collector. Generating the access token.
2024-02-15T06:47:57.288    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Getting the auth token url based on provided api_base_url
2024-02-15T06:47:57.288    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Getting the auth token url based on provided api_base_url
2024-02-15T06:47:59.494    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Puller Setup Terminated
2024-02-15T06:47:59.494    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,auditLogs#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-02-15T06:48:00.295    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> PrePull Started.
2024-02-15T06:48:00.295    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> User has specified 2023-11-11 20:10:02 as the datetime. Historical polling will consider this datetime for creating the default values.
2024-02-15T06:48:00.295 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Historic datetime in the persistence object and in the configuration are different. Updating the value in state with the user specified datetime.
2024-02-15T06:48:00.295 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Next poll will consider 2023-11-11 20:10:02 to now as the date range.
2024-02-15T06:48:00.296 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> PrePull Terminated
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Starting data collection every 60 seconds
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Pull Started
2024-02-15T06:48:00.296    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Fetching auditLogs from 2023-11-11T20:10:02Z
2024-02-15T06:48:00.297    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Requesting Wiz API for auditLogs
2024-02-15T06:48:01.886    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> successfully retried auditLogs from Wiz
2024-02-15T06:48:01.918    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:01.920    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:02.017    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> 500 issues delivered
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2024, 2, 15, 6, 47, 59, 126416), 'buffer_ids_with_duplication_risk': []}.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.295365) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Requesting Wiz API for auditLogs
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
2024-02-15T06:48:01.886    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> successfully retried auditLogs from Wiz
2024-02-15T06:48:01.918    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:01.919    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:01.920    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:02.017    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> 500 issues delivered
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2024, 2, 15, 6, 47, 59, 126416), 'buffer_ids_with_duplication_risk': []}.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.295365) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:02.055    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Requesting Wiz API for auditLogs
2024-02-15T06:48:32.320    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> successfully retried auditLogs from Wiz
2024-02-15T06:48:32.326    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:32.328    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:32.328    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:32.343    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> 343 issues delivered
2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 16, 1, 16, 19, 908609), 'buffer_ids_with_duplication_risk': []}.
2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Updating deduplication buffers content
2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707979680.295365):Number of requests made: 21; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 10343; Average of events per second: 322.735.
2024-02-15T06:48:32.345    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Pull Terminated
2024-02-15T06:48:32.345    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Data collection completed. Elapsed time: 32.050 seconds. Waiting for 27.950 second(s) until the next one

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707979680.295365):Number of requests made: 21; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 10343; Average of events per second: 322.735.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

CloudConfiguration Service

All events of this service are ingested into the table cspm.wiz.cloud_configuration.default

Issue service is based on the following GraphQL command:

query CloudConfigurationFindingsPage(
  $filterBy: ConfigurationFindingFilters
  $first: Int
  $after: String
  $orderBy: ConfigurationFindingOrder
) {
  configurationFindings(
    filterBy: $filterBy
    first: $first
    after: $after
    orderBy: $orderBy
  ) {
    nodes {
      id
      targetExternalId
      targetObjectProviderUniqueId
      firstSeenAt
      severity
      result
      status
      remediation
      resource {
        id
        providerId
        name
        nativeType
        type
        region
        subscription {
          id
          name
          externalId
          cloudProvider
        }
        projects {
          id
          name
          riskProfile {
            businessImpact
          }
        }
        tags {
          key
          value
        }
      }
      rule {
        id
        graphId
        name
        description
        remediationInstructions
        functionAsControl
      }
      securitySubCategories {
        id
        title
        category {
          id
          name
          framework {
            id
            name
          }
        }
      }
      ignoreRules {
        id
        name
        enabled
        expiredAt
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2024-02-15T06:47:57.290    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Puller Setup Started
2024-02-15T06:47:57.292    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> This is the first run of the collector. Generating the access token.
2024-02-15T06:47:57.292    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Getting the auth token url based on provided api_base_url
2024-02-15T06:47:57.292    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Using default Authentication Domain auth.wiz.io for fetching Access Token
2024-02-15T06:47:59.891    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Puller Setup Terminated
2024-02-15T06:47:59.891    INFO InputProcess::WizDataPullerSetup(Wiz_test_nikhil,wiz_data_puller#00001,cloudConfiguration#predefined) -> Setup for module <WizDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-02-15T06:48:00.290    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> PrePull Started.
2024-02-15T06:48:00.291    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> User has specified 2023-11-11 20:10:02 as the datetime. Historical polling will consider this datetime for creating the default values.
2024-02-15T06:48:00.291 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Historic datetime in the persistence object and in the configuration are different. Updating the value in state with the user specified datetime.
2024-02-15T06:48:00.291 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Next poll will consider 2023-11-11 20:10:02 to now as the date range.
2024-02-15T06:48:00.291 WARNING InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> PrePull Terminated
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Starting data collection every 60 seconds
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Pull Started
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Fetching cloudConfiguration from 2023-11-11T20:10:02Z
2024-02-15T06:48:00.292    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
2024-02-15T06:48:29.044    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> successfully retried cloudConfiguration from Wiz
2024-02-15T06:48:29.127    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Total number of events in this poll: 500
2024-02-15T06:48:29.128    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:29.128    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:29.129    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:29.238    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> 500 issues delivered
2024-02-15T06:48:29.239    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 23, 6, 46, 39, 979510), 'buffer_ids_with_duplication_risk': ['498e0032-f76f-58d8-935b-dc20546c4d77']}.
2024-02-15T06:48:29.239    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.290731) so far: Number of requests made: 1; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 500.
2024-02-15T06:48:29.239    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"}
2024-02-15T06:48:40.162    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> successfully retried cloudConfiguration from Wiz
2024-02-15T06:48:40.207    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:40.216    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:40.217    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:40.297    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> 500 issues delivered
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 24, 3, 29, 59, 755884), 'buffer_ids_with_duplication_risk': ['0e446865-0e54-567d-a3ab-f5bfedecefbc']}.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.290731) so far: Number of requests made: 2; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
2024-02-15T06:48:40.162    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> successfully retried cloudConfiguration from Wiz
2024-02-15T06:48:40.207    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Removing the duplicate issues if present
2024-02-15T06:48:40.216    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events
2024-02-15T06:48:40.217    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Delivering issues to the SDK
2024-02-15T06:48:40.297    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> 500 issues delivered
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'last_polled_timestamp': datetime.datetime(2023, 11, 11, 20, 10, 2, tzinfo=<UTC>), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2023, 11, 24, 3, 29, 59, 755884), 'buffer_ids_with_duplication_risk': ['0e446865-0e54-567d-a3ab-f5bfedecefbc']}.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1707979680.290731) so far: Number of requests made: 2; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 1000.
2024-02-15T06:48:40.299    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Requesting Wiz API for cloudConfiguration
2024-02-15T07:01:14.752    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707980301.689743):Number of requests made: 26; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 12779; Average of events per second: 73.841.
2024-02-15T07:01:14.752    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Pull Terminated

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-02-15T07:01:14.752    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,cloudConfiguration,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707980301.689743):Number of requests made: 26; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 12779; Average of events per second: 73.841.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_date_utc parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

Error type	Error ID	Error message	Cause	Solution

Error type	Error ID	Error message	Cause	Solution
`InitVariablesError`	`1`	`Devo tag is the required field for sending events to Devo. Specify it in collector definitions`	This error is raised when `devo_tag` property is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`2`	`Required setting. devo_tag is not of expected type: str`	This error is raised when `devo_tag` is defined in `collector_definitions.yaml` but the format is not `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`3`	`Optional setting, override_devo_tag not of expected type: str`	This error is raised when optional value `override_devo_tag` added in `config.json` is not of type `str`.	Edit the value of `override_devo_tag` in `config.json` so it is of type `str`. Or leave it empty so it takes the default value.
`InitVariablesError`	`4`	`GraphQL query is the required field for querying issues from Wiz. Specify it in collector definitions`	This error is raised when `graphql_query` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`5`	`Required setting. graphql_query is not of expected type: str`	This error is raised when `graphql_query` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`6`	`user_agent is the required field for passing in headers of Wiz API calls. Specify it in collector definitions`	This error is raised when `user_agent` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`7`	`Required setting. user_agent is not of expected type: str`	This error is raised when `user_agent` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`8`	`Optional setting, flatten_data not of expected type: bool`	This error is raised when the optional value `flatten_data` defined in `config.json` is not of type `bool`.	Edit the value of `flatten_data` in `config.json` so it is of type `bool`. You can also remove `override_flatten_data` parameter from `config.json` so it takes the default value.
`InitVariablesError`	`9`	`Optional setting, requests_per_second not of expected type: int`	This error is raised when the optional value `requests_per_second` defined in `config.json` is not of type `int`.	Edit the value of `requests_per_second` in `config.json` so it is of type `int`. Or leave it empty so it takes the default value.
`InitVariablesError`	`10`	`Required setting. requested_page_size_in_items is not of expected type: int`	This error is raised when `requested_page_size_in_items` defined in `collector_definitions.yaml` is not of type `int`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`11`	`access_token_timeout is the required field for checking if the token is expired. Specify it in collector definitions`	This error is raised when `access_token_timeout` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`12`	`Required setting. access_token_timeout is not of expected type: int`	This error is raised when `access_token_timeout` defined in `collector_definitions.yaml` is not of type `int`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`13`	`default_historic_days is the required field in case historic_date_utc is not specified. Specify it in collector definitions`	This error is raised when `default_historic_days` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`14`	`Required setting. default_historic_days is not of expected type: int`	This error is raised when `default_historic_days` defined in `collector_definitions.yaml` is not of type `int`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`15`	`api_url_regex is the required field for validating the base url. Specify it in collector definitions`	This error is raised when `api_url_regex` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`16`	`Required setting. api_url_regex is not of expected type: str`	This error is raised when `api_url_regex` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`17`	`historic_date_time_format is the required field for validating datetime format. Specify it in collector definitions`	This error is raised when `api_url_regex` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`18`	`Required setting. historic_date_time_format is not of expected type: str`	This error is raised when `api_url_regex` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`19`	`api_base_url not of expected type: str`	This error could be raised for two reasons: `api_base_url` defined in `collector_definitions.yaml` is not of type `str`. `override_api_base_url` defined in `config.json` is not of type `str`.	Solutions for both cases, respectively: This is an internal issue. Contact with Devo Support team. Edit the value of `override_api_base_url` in `config.json` so it is of type `str`. Or remove the parameter definition, so it takes the default value.
`InitVariablesError`	`20`	`api_base_url must match regex: <regex>`	This error could be raised for two reasons: `api_base_url` defined in `collector_definitions.yaml` does not match the required regex. `override_api_base_url` defined in `config.json` does not match the required regex.	Solutions for both cases, respectively: This is an internal issue. Contact with Devo Support team. Edit the value of `override_api_base_url` in `config.json` so it matches the indicated regex. Or remove the parameter definition so it takes the default value. If default value was used it is an internal issue, so contact with Devo Support team.
`InitVariablesError`	`21`	`Required setting, credentials not found in user configuration`	This error is raised when the required property `credentials` is not found in `config.json`.	Add `credentials` dictionary in `config.json`, including `client_id` and `client_secret` fields.
`InitVariablesError`	`22`	`Required setting, credentials not of expected type: dict`	This error is raised when `credentials` is defined in `config.json` but the format is not `dict`.	Edit the value of `credentials` in `config.json` so it is of type `dict`.
`InitVariablesError`	`23`	`Required setting, client_id not found in user configuration`	This error is raised when the required property `client_id` is not found in `config.json`, into `credentials` dictionary.	Add `client_id` property in `config.json`, into `credentials` dictionary.
`InitVariablesError`	`24`	`Required setting, client_id not of expected type: str`	This error is raised when `client_id` is defined in `config.json` but the format is not `str`.	Edit the value of `client_id` in `config.json`, into `credentials` dictionary, so it is of type `str`.
`InitVariablesError`	`25`	`Required setting, client_secret not found in user configuration`	This error is raised when the required property `client_secret` is not found in `config.json`, into `credentials` dictionary.	Add `client_secret` property in `config.json`, into `credentials` dictionary.
`InitVariablesError`	`26`	`Required setting, client_secret not of expected type: str`	This error is raised when `client_secret` is defined in `config.json` but the format is not `str`.	Edit the value of `client_secret` in `config.json`, into `credentials` dictionary, so it is of type `str`.
`InitVariablesError`	`27`	`Required setting, historic_date_utc not of expected type: str`	This error is raised when the optional value `historic_date_utc` defined in `config.json` is not of type `str`.	Edit the value of `historic_date_utc` in `config.json` so it is of type `str`. Or leave it empty so the collector starts pulling data N days ago at the current time.
`InitVariablesError`	`28`	`Time format for historic date must be <time_format>. e.g. 2022-02-15T14:32:33.043Z`	This error is raised when the optional value `historic_date_utc` defined in `config.json` does not match the indicated format.	Make the value of `historic_date_utc` in `config.json` match the indicated format. Or leave it empty so the collector starts pulling data N days ago at the current time.
`InitVariablesError`	`29`	`historic datetime cannot be greater than the present UTC time`	This error is raised when the optional value `historic_date_utc` defined in `config.json` is higher than the current time. Time is indicated in UTC time.	Make the value of `historic_date_utc` in `config.json` be a past value in UTC time. Or leave it empty so the collector starts pulling data N days ago at the current time.
`SetupError`	`100`	`Error occurred while requesting access token from the Wiz server. Error message: <error_message>`	This error is raised when a generic error occurs during the request to get the token to authenticate the collector in the API.	This is an internal issue. Contact with Devo Support team.
`SetupError`	`101`	`The credentials provided in the config file are incorrect. Please provide the correct credentials.` `Status code: 401` `Error type: <error_type>` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are not valid. (401 Unauthorized error)	Add the correct `client_id` and `client_secret` fields in `config.json`, at `credentials` property.
`SetupError`	`102`	`The credentials provided in the config file does not have necessary permissions to create access token.` `Status code: 403` `Error type: <error_type>` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but the credentials used to access the API have no permissions to create a token. (403 Forbidden error)	Add some credentials in `config.json` with enough privileges to create a token from the API.
`SetupError`	`103`	`The requested URL <URL> is not found. The URL may have been deprecated.` `Status code: 404` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but the authentication endpoint that is being requested to get a token is not found.	This is an internal issue. Contact with Devo Support team.
`SetupError`	`104`	`Unexpected error occurred while getting access token from the Wiz server` `Status code: <status_code>` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but an unexpected response has been returned from the API.	This is an internal issue. Contact with Devo Support team.
`SetupError`	`105`	`The credentials does not have valid permissions to fetch issues from the Wiz server`	This error is raised when the credentials provided in `config.json` are valid, but the credentials used have no permissions to access the API endpoint to get issues.	Check that the provided credentials have enough permission to retrieve issues from the API.
`SetupError`	`106`	`Failed to check if the provided credentials have valid permissions.` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but there was an error in the process of checking if the provided credentials have permission to get issues.	This is an internal issue. Contact with Devo Support team.
`PullError`	`300`	`Error occurred while requesting issues from the Wiz server. Error message: <error_message>`	This error is raised when a generic error occurs during the request to get issues from the API.	This is an internal issue. Contact with Devo Support team.
`PullError`	`301`	`The token used to make this request is not valid anymore.` `Status code: 401` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is not valid anymore.	Check if the credentials need to be renewed. Add some valid credentials in `config.json`. If credentials are still valid, contact with Devo Support team.
`PullError`	`302`	`The access token does not have necessary permissions to fetch issues from Wiz.` `Status code: 403` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but it lost permissions to get issues from API.	Check what happened with credential permissions. Add some credentials in `config.json` with permissions to get issues. If credentials are valide and still have permissions, contact with Devo Support team.
`PullError`	`303`	`The requested URL <URL> is not found. The URL may have been depreciated` `Status code: 404` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but cannot find the endpoints to get issues.	This is an internal issue. Contact with Devo Support team.
`PullError`	`304`	`The server has returned <status_code> status code. The server may not be available for fetching issues. Try after sometime. Error message from server: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but there has been an error on Wiz's API.	The error is on Wiz’s side. Wiz can be contacted for more info. It should work again when the incident at Wiz is solved.
`PullError`	`305`	`Unexpected error occurred while getting issues from the Wiz server` `Status code: <status_code>` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but there has been an unexpected return from the API.	This is an internal issue. Contact with Devo Support team.
`PullError`	`306`	`After <retry_count> retries still getting the too many requests error.`	This error is raised when the token being used to make requests to the API is valid, but we are constantly receiving a `429` error response (too many requests)	Check throttle limitations on Wiz API and change the value of `request_period_in_seconds` and xxxxxxxx from `config.json` to adapt to it.

Custom Service

All events of this service are ingested into the table my.app.wiz.custom_query by default. You need to provide override_tag if you need to change it.

CloudEventsQuery
- Override Devo tag : cspm.wiz.cloud_event.default
- filter_by_time_key: timestamp
- response_time_key: timestamp
- filter_by: { "origin": { "equals": "WIZ_SENSOR" } }

filter_by is just an example and can change as per requirement. All above fields need to be in user config.

query CloudEventsQuery(
  $first: Int
  $after: String
  $filterBy: CloudEventFilters
) {
  cloudEvents(first: $first, after: $after, filterBy: $filterBy) {
    __typename
    totalCount
    nodes {
      ... on CloudEvent {
        id
        name
        kind
        origin
        severity
        externalId
        externalName
        cloudPlatform
        timestamp
        cloudNativeService
        category
        actor {
          id
          externalId
          providerUniqueId
          type
          cloudAccount {
            id
          }
          isExternalCloudAccount
          friendlyName
          name
          email
          userAgent
        }
        actorIP
        actorIPMeta {
          country
          countryCode
          city
          reputation
          reputationSource
          reputationDescription
        }
        isForeignActorIP
        subjectResource {
          id
          externalId
          providerUniqueId
          type
          nativeType
          name
          hostname
          cloudAccount {
            id
            externalId
            name
            cloudProvider
            linkedProjects {
              name
            }
          }
          region
          tags
          openToAllInternet
          hasSensitiveData
          kubernetesCluster {
            id
            name
            type
          }
          kubernetesNamespace {
            id
            name
            type
          }
          kubernetesFlavor
          containerService {
            id
            name
          }
        }
        cloudProviderUrl
        path
        hash
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

systemActivitiesQuery
- Override Devo tag : cspm.wiz.system_activity.default
- filter_by_time_key: createdAt
- response_time_key: createdAt
- filter_by: {}

filter_by is just an example and can change as per requirement. All above fields need to be in user config.

query systemActivitiesQuery(
  $first: Int
  $after: String
  $filterBy: SystemActivityFilters
) {
  systemActivities(first: $first, after: $after, filterBy: $filterBy) {
    __typename
    totalCount
    nodes {
      ... on SystemActivity {
        id
        name
        triggerType
        triggeredBy {
          ... on SystemActivityUserSnapshot {
            id
            name
          }
        }
        triggeredBy {
          ... on SystemActivitySystemTrigger {
            id
          }
        }
        triggeredBy {
          ... on SystemActivityServiceAccountSnapshot {
            id
            name
          }
        }
        createdAt
        startedAt
        endedAt
        status
        statusInfo
        summary
        groupId
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2024-07-12T14:44:51.388    INFO InputProcess::WizCustomDataPullerSetup(unknown,wiz_data_puller#3454335,custom_query#custom) -> Puller Setup Started
2024-07-12T14:44:51.388    INFO InputProcess::WizCustomDataPullerSetup(unknown,wiz_data_puller#3454335,custom_query#custom) -> This is the first run of the collector. Generating the access token.
2024-07-12T14:44:51.388    INFO InputProcess::WizCustomDataPullerSetup(unknown,wiz_data_puller#3454335,custom_query#custom) -> Getting the auth token url based on provided api_base_url
2024-07-12T14:44:52.690    INFO InputProcess::WizCustomDataPullerSetup(unknown,wiz_data_puller#3454335,custom_query#custom) -> successfully generated new access token
2024-07-12T14:44:53.548    INFO InputProcess::WizCustomDataPullerSetup(unknown,wiz_data_puller#3454335,custom_query#custom) -> The credentials provided in the configuration have required permissions to request issues from Wiz server
2024-07-12T14:44:53.549    INFO InputProcess::WizCustomDataPullerSetup(unknown,wiz_data_puller#3454335,custom_query#custom) -> Puller Setup Terminated
2024-07-12T14:44:53.549    INFO InputProcess::WizCustomDataPullerSetup(unknown,wiz_data_puller#3454335,custom_query#custom) -> Setup for module <WizCustomDataPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-07-12T14:44:54.393    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> PrePull Started.
2024-07-12T14:44:54.396    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> User has specified 2024-07-05 06:03:52 as the datetime. Historical polling will consider this datetime for creating the default values.
2024-07-12T14:44:54.396    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2024, 7, 5, 6, 3, 52), 'last_polled_timestamp': datetime.datetime(2024, 7, 5, 6, 3, 52), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-07-12T14:44:54.398 WARNING InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2024, 7, 5, 6, 3, 52), 'last_polled_timestamp': datetime.datetime(2024, 7, 5, 6, 3, 52), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []}
2024-07-12T14:44:54.398    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> PrePull Terminated
2024-07-12T14:44:54.399    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Starting data collection every 300 seconds
2024-07-12T14:44:54.399    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Pull Started
2024-07-12T14:44:54.401    INFO OutputProcess::ConsoleSender(internal_senders,console_sender_0) -> {"message_timestamp": "2024-07-12 09:14:54.399", "message_tag": "devo.collectors.out.local.info", "message_content": "{\"msg\": \"Starting data collection every 300 seconds\", \"time\": \"2024-07-12T09:14:54.399167Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"job_id\": \"unknown\", \"input_name\": \"wiz_data_puller\", \"service_name\": \"custom_query\", \"module_name\": \"WizCustomDataPuller\"}"}
2024-07-12T14:44:54.401    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Fetching custom_query from 2024-07-05T00:33:52Z
2024-07-12T14:44:54.401    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Requesting Wiz API for custom_query
2024-07-12T14:44:58.123    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> successfully retried custom_query from Wiz
2024-07-12T14:44:58.136    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Total number of events in this poll: 500
2024-07-12T14:44:58.202    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Flatten data is set to False. Adding 'devo_pulling_id' to events
2024-07-12T14:44:58.202    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> Delivering issues to the SDK
2024-07-12T14:44:58.236    INFO InputProcess::WizCustomDataPuller(wiz_data_puller,3454335,custom_query,custom) -> 500 issues delivered

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-02-15T06:48:32.344    INFO InputProcess::WizDataPuller(wiz_data_puller,00001,auditLogs,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1707979680.295365):Number of requests made: 21; Number of events received: 500; Number of duplicated events filtered out: 0; Number of events generated and sent: 10343; Average of events per second: 322.735.

The value @devo_pulling_id is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull action in Devo’s search window.

Note that a Partial Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial reference.

(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the historical_date_utc parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

Error type	Error ID	Error message	Cause	Solution

Error type	Error ID	Error message	Cause	Solution
`InitVariablesError`	`1`	`Devo tag is the required field for sending events to Devo. Specify it in collector definitions`	This error is raised when `devo_tag` property is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`2`	`Required setting. devo_tag is not of expected type: str`	This error is raised when `devo_tag` is defined in `collector_definitions.yaml` but the format is not `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`3`	`Optional setting, override_devo_tag not of expected type: str`	This error is raised when optional value `override_devo_tag` added in `config.json` is not of type `str`.	Edit the value of `override_devo_tag` in `config.json` so it is of type `str`. Or leave it empty so it takes the default value.
`InitVariablesError`	`4`	`GraphQL query is the required field for querying issues from Wiz. Specify it in collector definitions`	This error is raised when `graphql_query` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`5`	`Required setting. graphql_query is not of expected type: str`	This error is raised when `graphql_query` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`6`	`user_agent is the required field for passing in headers of Wiz API calls. Specify it in collector definitions`	This error is raised when `user_agent` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`7`	`Required setting. user_agent is not of expected type: str`	This error is raised when `user_agent` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`8`	`Optional setting, flatten_data not of expected type: bool`	This error is raised when the optional value `flatten_data` defined in `config.json` is not of type `bool`.	Edit the value of `flatten_data` in `config.json` so it is of type `bool`. You can also remove `override_flatten_data` parameter from `config.json` so it takes the default value.
`InitVariablesError`	`9`	`Optional setting, requests_per_second not of expected type: int`	This error is raised when the optional value `requests_per_second` defined in `config.json` is not of type `int`.	Edit the value of `requests_per_second` in `config.json` so it is of type `int`. Or leave it empty so it takes the default value.
`InitVariablesError`	`10`	`Required setting. requested_page_size_in_items is not of expected type: int`	This error is raised when `requested_page_size_in_items` defined in `collector_definitions.yaml` is not of type `int`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`11`	`access_token_timeout is the required field for checking if the token is expired. Specify it in collector definitions`	This error is raised when `access_token_timeout` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`12`	`Required setting. access_token_timeout is not of expected type: int`	This error is raised when `access_token_timeout` defined in `collector_definitions.yaml` is not of type `int`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`13`	`default_historic_days is the required field in case historic_date_utc is not specified. Specify it in collector definitions`	This error is raised when `default_historic_days` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`14`	`Required setting. default_historic_days is not of expected type: int`	This error is raised when `default_historic_days` defined in `collector_definitions.yaml` is not of type `int`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`15`	`api_url_regex is the required field for validating the base url. Specify it in collector definitions`	This error is raised when `api_url_regex` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`16`	`Required setting. api_url_regex is not of expected type: str`	This error is raised when `api_url_regex` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`17`	`historic_date_time_format is the required field for validating datetime format. Specify it in collector definitions`	This error is raised when `api_url_regex` is not found in `collector_definitions.yaml`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`18`	`Required setting. historic_date_time_format is not of expected type: str`	This error is raised when `api_url_regex` defined in `collector_definitions.yaml` is not of type `str`.	This is an internal issue. Contact with Devo Support team.
`InitVariablesError`	`19`	`api_base_url not of expected type: str`	This error could be raised for two reasons: `api_base_url` defined in `collector_definitions.yaml` is not of type `str`. `override_api_base_url` defined in `config.json` is not of type `str`.	Solutions for both cases, respectively: This is an internal issue. Contact with Devo Support team. Edit the value of `override_api_base_url` in `config.json` so it is of type `str`. Or remove the parameter definition, so it takes the default value.
`InitVariablesError`	`20`	`api_base_url must match regex: <regex>`	This error could be raised for two reasons: `api_base_url` defined in `collector_definitions.yaml` does not match the required regex. `override_api_base_url` defined in `config.json` does not match the required regex.	Solutions for both cases, respectively: This is an internal issue. Contact with Devo Support team. Edit the value of `override_api_base_url` in `config.json` so it matches the indicated regex. Or remove the parameter definition so it takes the default value. If default value was used it is an internal issue, so contact with Devo Support team.
`InitVariablesError`	`21`	`Required setting, credentials not found in user configuration`	This error is raised when the required property `credentials` is not found in `config.json`.	Add `credentials` dictionary in `config.json`, including `client_id` and `client_secret` fields.
`InitVariablesError`	`22`	`Required setting, credentials not of expected type: dict`	This error is raised when `credentials` is defined in `config.json` but the format is not `dict`.	Edit the value of `credentials` in `config.json` so it is of type `dict`.
`InitVariablesError`	`23`	`Required setting, client_id not found in user configuration`	This error is raised when the required property `client_id` is not found in `config.json`, into `credentials` dictionary.	Add `client_id` property in `config.json`, into `credentials` dictionary.
`InitVariablesError`	`24`	`Required setting, client_id not of expected type: str`	This error is raised when `client_id` is defined in `config.json` but the format is not `str`.	Edit the value of `client_id` in `config.json`, into `credentials` dictionary, so it is of type `str`.
`InitVariablesError`	`25`	`Required setting, client_secret not found in user configuration`	This error is raised when the required property `client_secret` is not found in `config.json`, into `credentials` dictionary.	Add `client_secret` property in `config.json`, into `credentials` dictionary.
`InitVariablesError`	`26`	`Required setting, client_secret not of expected type: str`	This error is raised when `client_secret` is defined in `config.json` but the format is not `str`.	Edit the value of `client_secret` in `config.json`, into `credentials` dictionary, so it is of type `str`.
`InitVariablesError`	`27`	`Required setting, historic_date_utc not of expected type: str`	This error is raised when the optional value `historic_date_utc` defined in `config.json` is not of type `str`.	Edit the value of `historic_date_utc` in `config.json` so it is of type `str`. Or leave it empty so the collector starts pulling data N days ago at the current time.
`InitVariablesError`	`28`	`Time format for historic date must be <time_format>. e.g. 2022-02-15T14:32:33.043Z`	This error is raised when the optional value `historic_date_utc` defined in `config.json` does not match the indicated format.	Make the value of `historic_date_utc` in `config.json` match the indicated format. Or leave it empty so the collector starts pulling data N days ago at the current time.
`InitVariablesError`	`29`	`historic datetime cannot be greater than the present UTC time`	This error is raised when the optional value `historic_date_utc` defined in `config.json` is higher than the current time. Time is indicated in UTC time.	Make the value of `historic_date_utc` in `config.json` be a past value in UTC time. Or leave it empty so the collector starts pulling data N days ago at the current time.
`SetupError`	`100`	`Error occurred while requesting access token from the Wiz server. Error message: <error_message>`	This error is raised when a generic error occurs during the request to get the token to authenticate the collector in the API.	This is an internal issue. Contact with Devo Support team.
`SetupError`	`101`	`The credentials provided in the config file are incorrect. Please provide the correct credentials.` `Status code: 401` `Error type: <error_type>` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are not valid. (401 Unauthorized error)	Add the correct `client_id` and `client_secret` fields in `config.json`, at `credentials` property.
`SetupError`	`102`	`The credentials provided in the config file does not have necessary permissions to create access token.` `Status code: 403` `Error type: <error_type>` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but the credentials used to access the API have no permissions to create a token. (403 Forbidden error)	Add some credentials in `config.json` with enough privileges to create a token from the API.
`SetupError`	`103`	`The requested URL <URL> is not found. The URL may have been deprecated.` `Status code: 404` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but the authentication endpoint that is being requested to get a token is not found.	This is an internal issue. Contact with Devo Support team.
`SetupError`	`104`	`Unexpected error occurred while getting access token from the Wiz server` `Status code: <status_code>` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but an unexpected response has been returned from the API.	This is an internal issue. Contact with Devo Support team.
`SetupError`	`105`	`The credentials does not have valid permissions to fetch issues from the Wiz server`	This error is raised when the credentials provided in `config.json` are valid, but the credentials used have no permissions to access the API endpoint to get issues.	Check that the provided credentials have enough permission to retrieve issues from the API.
`SetupError`	`106`	`Failed to check if the provided credentials have valid permissions.` `Error message: <error_message>`	This error is raised when the credentials provided in `config.json` are valid, but there was an error in the process of checking if the provided credentials have permission to get issues.	This is an internal issue. Contact with Devo Support team.
`PullError`	`300`	`Error occurred while requesting issues from the Wiz server. Error message: <error_message>`	This error is raised when a generic error occurs during the request to get issues from the API.	This is an internal issue. Contact with Devo Support team.
`PullError`	`301`	`The token used to make this request is not valid anymore.` `Status code: 401` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is not valid anymore.	Check if the credentials need to be renewed. Add some valid credentials in `config.json`. If credentials are still valid, contact with Devo Support team.
`PullError`	`302`	`The access token does not have necessary permissions to fetch issues from Wiz.` `Status code: 403` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but it lost permissions to get issues from API.	Check what happened with credential permissions. Add some credentials in `config.json` with permissions to get issues. If credentials are valide and still have permissions, contact with Devo Support team.
`PullError`	`303`	`The requested URL <URL> is not found. The URL may have been depreciated` `Status code: 404` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but cannot find the endpoints to get issues.	This is an internal issue. Contact with Devo Support team.
`PullError`	`304`	`The server has returned <status_code> status code. The server may not be available for fetching issues. Try after sometime. Error message from server: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but there has been an error on Wiz's API.	The error is on Wiz’s side. Wiz can be contacted for more info. It should work again when the incident at Wiz is solved.
`PullError`	`305`	`Unexpected error occurred while getting issues from the Wiz server` `Status code: <status_code>` `Error message: <error_message>`	This error is raised when the token being used to make requests to the API is valid, but there has been an unexpected return from the API.	This is an internal issue. Contact with Devo Support team.
`PullError`	`306`	`After <retry_count> retries still getting the too many requests error.`	This error is raised when the token being used to make requests to the API is valid, but we are constantly receiving a `429` error response (too many requests)	Check throttle limitations on Wiz API and change the value of `request_period_in_seconds` and xxxxxxxx from `config.json` to adapt to it.

Collector operations

This section is intended to explain how to proceed with specific operations of this collector.

Initialization

The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.

A successful run has the following output messages for the initializer module:

INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 1000, max_size_in_mb: 1024, max_wrap_size_in_items: 100
INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-eu.devo.io', 'port': 443, ...}}}
INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s)
INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s)
INFO OutputProcess::MainThread -> Process started
INFO InputProcess::MainThread -> Process Started
INFO InputProcess::MainThread -> InitVariables Started
INFO InputProcess::MainThread -> Validating variables in collector definitions Started
INFO InputProcess::MainThread -> Flatten data is not provided in the config.yaml. Considering the flatten data from collector definitions
INFO InputProcess::MainThread -> Validating collector Variables is terminated
INFO InputProcess::MainThread -> Initialization of api_base_url has started.
INFO InputProcess::MainThread -> api_base_url has been initialized
INFO InputProcess::MainThread -> Initialization of credentials has started.
INFO InputProcess::MainThread -> credentials have been initialized.
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO InputProcess::MainThread -> InitVariables Terminated
INFO InputProcess::MainThread -> InputThread(wiz_data_puller,111) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> ServiceThread(wiz_data_puller,111,issues,predefined) - Starting thread (execution_period=120s)
INFO InputProcess::MainThread -> WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Starting thread
INFO InputProcess::MainThread -> WizDataPuller(wiz_data_puller,111,issues,predefined) - Starting thread
WARNING InputProcess::WizDataPuller(wiz_data_puller,111,issues,predefined) -> Waiting until setup will be executed
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> This is the first run of collector. Generating the access token
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Getting the auth token url based on provided api_base_url
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Using default Authentication Domain auth.wiz.io for fetching Access Token
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io
INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread
INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds)
INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread
INFO InputProcess::MainThread -> [GC] global: 36.7% -> 36.7%, process: RSS(26.93MiB -> 27.97MiB), VMS(334.43MiB -> 334.67MiB)
INFO OutputProcess::MainThread -> [GC] global: 36.7% -> 36.3%, process: RSS(26.68MiB -> 28.61MiB), VMS(910.71MiB -> 910.71MiB)
INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-eu.devo.io:443", ...}

Events delivery and Devo ingestion

The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.

A successful run has the following output messages for the initializer module:

INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds)
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True}
INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)

By default, these information traces will be displayed every 10 minutes.

Sender services

The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal, standard, and lookup). This collector uses the following Sender Services:

Sender services	Description

Sender services	Description
`internal_senders`	In charge of delivering internal metrics to Devo such as logging traces or metrics.
`standard_senders`	In charge of delivering pulled events to Devo.

Sender statistics

Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:

Logging trace	Description

Logging trace

Description

Number of available senders: 1

Displays the number of concurrent senders available for the given Sender Service.

sender manager internal queue size: 0

Displays the items available in the internal sender queue.

This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders.

Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 21 (elapsed 0.007 seconds)

Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:

44 events were sent to Devo since the collector started.
The last checkpoint timestamp was 2022-06-28 10:39:22.511671+00:00.
21 events where sent to Devo between the last UTC checkpoint and now.
Those 21 events required 0.007 seconds to be delivered.

By default these traces will be shown every 10 minutes.

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB)
INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)

Differences between RSS and VMS memory usage:

RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using

Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.

To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.

For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.

Change log

Release	Released on	Release type	Details	Recommendations

Release	Released on	Release type	Details	Recommendations
`1.7.0`	Jul 12, 2024	status:FEATUREstatus:IMPROVEMENTS	New Features Added custom_query service Improvements Updated the DCSDK to v1.12.2 Added new sender for relay in house + TLS Added persistence functionality for gzip sending buffer Added Automatic activation of gzip sending Improved behaviour when persistence fails Upgraded DevoSDK dependency Fixed console log encoding Restructured python classes Improved behavior with non-utf8 characters Decreased defaut size value for internal queues (Redis limitation, from 1GiB to 256MiB) New persistence format/structure (compression in some cases) Removed dmesg execution (It was invalid for docker execution) DevoSDK has been updated to version 5.4.0 Updated dcsdk-docker-base-image to 1.3.0	`Recommended version`
`1.6.1`	Mar 13, 2024	status:BUG FIX	Bug Fixes Updating the issues query to include extra fields.	`Upgrade`
`v1.6.0`	Feb 29, 2024	status:BUG FIXstatus:IMPROVEMENTS	Bug Fixes Updated the issues query to v2 for it to work properly. Improvements Added the DCSDK auto update feature. Updated the DCSDK from 1.10.3 to 1.11.0.	`Upgrade`
`v1.5.0`	Feb 15, 2024	status:FEATUREstatus:IMPROVEMENTS	New Features Added following three new services: Vulnerabilities Audit Logs Cloud Configuration Findings Improvements Upgraded DCSDK from 1.10.2 to 1.10.3 .	`Upgrade`
`v1.4.0`	Nov 20, 2023	status:FEATUREstatus:IMPROVEMENTS	New Features Added extra filters for events: type: Filter by Issue type. You can specify multiple values in an array. Possible values: ["TOXIC_COMBINATION", "THREAT_DETECTION", "CLOUD_CONFIGURATION"] Improvements Upgraded DCSDK from 1.9.2 to 1.10.2 Added input metrics Modified ouutput metrics Updated DevoSDK to version 5.1.6 Standardized exception messages for traceability Added more detail in queue statistics Updated PythonSDK to version 5.0.7 Introduced pyproject.toml Added requirements.dev.txt Fixed error in pyproject.toml related to project scripts endpoint	`Recommended`
`v1.3.0`	Oct 30, 2023	status:BUG FIXstatus:IMPROVEMENTS	Improvements: Upgraded DCSDK from 1.9.1 to 1.9.2 upgraded dependencies Bug Fix: Remove actions from service tables	`Recommended`
`v1.2.0`	Aug 23, 2023	status:IMPROVEMENTS	Improvements: Upgraded DCSDK from 1.3.0 to 1.9.1 Store lookup instances into DevoSender to avoid creation of new instances for the same lookup Ensure service_config is a dict into templates Ensure special characters are properly sent to the platform Changed log level to some messages from info to debug Changed some wrong log messages Upgraded some internal dependencies Changed queue passed to setup instance constructor Added log traces for knowing the execution environment status (debug mode) Fixes in the current puller template version Improved log trace details when runtime exceptions happen Refactored source code structure New “templates” functionality Functionality for detecting some system signals for starting the controlled stopping Input objects sends again the internal messages to devo.collectors.out table Upgraded DevoSDK to version 3.6.4 to fix a bug related to a connection loss with Devo Refactored source code structure Changed way of executing the controlled stopping Minimized probabilities of suffering a DevoSDK bug related to “sender” to be null Ability to validate collector setup and exit without pulling any data Ability to store in the persistence the messages that couldn’t be sent after the collector stopped Ability to send messages from the persistence when the collector starts and before the puller begins working Ensure special characters are properly sent to the platform Added a lock to enhance sender object Added new class attrs to the __setstate__ and __getstate__ queue methods Fix sending attribute value to the __setstate__ and __getstate__ queue methods Added log traces when queues are full and have to wait Added log traces of queues time waiting every minute in debug mode Added method to calculate queue size in bytes Block incoming events in queues when there are no space left Send telemetry events to Devo platform Upgraded internal Python dependency Redis to v4.5.4 Upgraded internal Python dependency DevoSDK to v5.1.3 Fixed obfuscation not working when messages are sent from templates New method to figure out if a puller thread is stopping Upgraded internal Python dependency DevoSDK to v5.0.6 Improved logging on messages/bytes sent to Devo platform Fixed wrong bytes size calculation for queues New functionality to count bytes sent to Devo Platform (shown in console log) Upgraded internal Python dependency DevoSDK to v5.0.4 Fixed bug in persistence management process, related to persistence reset Aligned source code typing to be aligned with Python 3.9.x Inject environment property from user config Obfuscation service can be now configured from user config and module definition Obfuscation service can now obfuscate items inside arrays Ensure special characters are properly sent to the platform The resilience has been improved with a new feature that restart the collector when the Devo connections is lost and it cannot be recovered. When an exception is raised by the Collector Setup, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied. When an exception is raised by the Collector Pull method, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied. When an exception is raised by the Collector pre-pull method, the collector retries after 30 seconds. No maximum retries are applied. Changed log level to some messages from info to debug Changed some wrong log messages Upgraded some internal dependencies Changed queue passed to setup instance constructor	`Upgrade`
`v1.1.1`	Oct 7, 2022	status:BUG FIX	Bug fixes: Force using always UTC timezone for all date time operations.	`Recommended version`
`v1.1.0`	Jul 21, 2022	status:FEATURE	New features: Wiz’s new authentication via `Cognito` is now available. Former authentication using `Auth0` is also still compatible.	`Recommended version`
`v1.0.0`	Jul 1, 2022	status:FEATURE	New features: Wiz issues	`Upgrade`

Devo latest

Wiz collector

Overview

Data sources

Devo collector features

Flattening preprocessing

How to enable the collection in the vendor

Minimal requirements to follow this guide

Minimum configuration required for basic pulling

Accepted authentication methods

Run the collector

Collector service details

Issue Service

Setup output

Puller output

Vulnerability Service

Setup output

Puller output

AuditLogs Service

Setup output

Puller output

CloudConfiguration Service

Setup output

Puller output

Custom Service

Setup output

Puller output

Collector operations

Initialization

Events delivery and Devo ingestion

Sender services

Sender statistics

Change log

New Features

Improvements

New Features

Improvements

Related content