- Created by Juan Tomás Alonso Nieto, last modified on Feb 16, 2024
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 18 Next »
Overview
Wiz is a cloud infrastructure security tool that provides organizations with an in-depth contextual risk assessment. Wiz’s agentless solution builds inventory, and scans for varied risk factors such as vulnerabilities, excessive permissions, malware, exposed secrets, practical exposure, and more, and prioritizes the alerts for the security teams based on the likelihood to be exploited and potential business impact.
The Devo Wiz collector allows customers to retrieve Wiz cloud security issues into Devo to query, correlate, analyze, and visualize to enable Enterprise IT and Cybersecurity teams to take the most impactful decisions at the petabyte scale. The collector processes the Wiz API responses and sends them to the Devo platform, which then categorizes all data received on tables along rows and columns in your Devo domain.
Data sources
Data source | Description | API Endpoint | Collector service name | Devo table | Available from release |
---|---|---|---|---|---|
Issues | An issue in wiz is a vulnerability that is detected in the cloud infrastructure |
|
|
|
|
Vulnerability | Vulnerabilities are weaknesses in computer systems that can be exploited by malicious attackers. Whether they are caused by bugs or design flaws, vulnerabilities can allow attackers to execute code in an environment or elevate privileges. |
|
|
|
|
Audit Logs | The Audit Log records key events in Wiz, such as login, logout, and user update. The Audit Log is primarily used to investigate potentially suspicious activity or diagnose and troubleshoot errors. |
|
|
|
|
Cloud Configuration Findings | This returns the problems with configurations and the remediation solutions for the same. |
|
|
|
|
Devo collector features
Feature | Details |
---|---|
Allow parallel downloading ( |
|
Running environments |
|
Populated Devo events |
|
Flattening preprocessing |
|
Flattening preprocessing
In order to improve the data exploitation and enrichment, this collector applies some flattening actions to the collected data before delivering it to Devo:
Data source | Collector service | Optional | Flattening details |
---|---|---|---|
Issues |
|
|
|
Vulnerabilities |
|
|
|
Audit Logs |
|
|
|
Cloud Configuration Findings |
|
|
|
How to enable the collection in the vendor
Minimal requirements to follow this guide
In order to retrieve the data, the following details will be required from your Wiz instance.
Instance domain | Wiz domain of your cloud instance where the collector will make the requests. |
---|---|
Client ID | Wiz user ID. |
Client secret | Wiz user passwords. |
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.
This minimum configuration refers exclusively to the specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting | Details |
---|---|
| By default, the base URLis |
| User Client ID to authenticate to the service. |
| User Secret Key to authenticate to the service. |
Accepted authentication methods
The following are the accepted authentication methods for this collector.
Authentication method | Client ID | Client secret |
---|---|---|
Basic authentication | REQUIRED | REQUIRED |
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector service details
Issue Service
All events of this service are ingested into the table cspm.wiz.issues.default
Issue service is based on the following GraphQL command:
query IssuesTable( $filterBy: IssueFilters $first: Int $after: String $orderBy: IssueOrder ) { issues( filterBy: $filterBy first: $first after: $after orderBy: $orderBy ) { nodes { ...IssueDetails } pageInfo { hasNextPage endCursor } totalCount informationalSeverityCount lowSeverityCount mediumSeverityCount highSeverityCount criticalSeverityCount uniqueEntityCount } } fragment IssueDetails on Issue { id control { id name query securitySubCategories { id title category { id name framework { id name } } } } createdAt updatedAt projects { id name slug businessUnit riskProfile { businessImpact } } status severity entity { id name type } entitySnapshot { id type nativeType name subscriptionId subscriptionExternalId subscriptionName resourceGroupId resourceGroupExternalId region cloudPlatform cloudProviderURL providerId status tags subscriptionTags } note serviceTicket { externalId name url } serviceTickets { externalId name url action { id type } } }
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> successfully generated new access token INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Terminated INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Setup for module <WizDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Started. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} WARNING InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Terminated 2INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Started INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 5 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Updating deduplication buffers content INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Terminated INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
Note that a Partial
Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial
reference.
(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
Vulnerability Service
Issue Service
All events of this service are ingested into the table cspm.wiz.issues.default
Issue service is based on the following GraphQL command:
query IssuesTable( $filterBy: IssueFilters $first: Int $after: String $orderBy: IssueOrder ) { issues( filterBy: $filterBy first: $first after: $after orderBy: $orderBy ) { nodes { ...IssueDetails } pageInfo { hasNextPage endCursor } totalCount informationalSeverityCount lowSeverityCount mediumSeverityCount highSeverityCount criticalSeverityCount uniqueEntityCount } } fragment IssueDetails on Issue { id control { id name query securitySubCategories { id title category { id name framework { id name } } } } createdAt updatedAt projects { id name slug businessUnit riskProfile { businessImpact } } status severity entity { id name type } entitySnapshot { id type nativeType name subscriptionId subscriptionExternalId subscriptionName resourceGroupId resourceGroupExternalId region cloudPlatform cloudProviderURL providerId status tags subscriptionTags } note serviceTicket { externalId name url } serviceTickets { externalId name url action { id type } } }
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> successfully generated new access token INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Terminated INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Setup for module <WizDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Started. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} WARNING InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Terminated 2INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Started INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 5 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Updating deduplication buffers content INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Terminated INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
Note that a Partial
Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial
reference.
(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
AuditLogs Service
All events of this service are ingested into the table cspm.wiz.issues.default
Issue service is based on the following GraphQL command:
query IssuesTable( $filterBy: IssueFilters $first: Int $after: String $orderBy: IssueOrder ) { issues( filterBy: $filterBy first: $first after: $after orderBy: $orderBy ) { nodes { ...IssueDetails } pageInfo { hasNextPage endCursor } totalCount informationalSeverityCount lowSeverityCount mediumSeverityCount highSeverityCount criticalSeverityCount uniqueEntityCount } } fragment IssueDetails on Issue { id control { id name query securitySubCategories { id title category { id name framework { id name } } } } createdAt updatedAt projects { id name slug businessUnit riskProfile { businessImpact } } status severity entity { id name type } entitySnapshot { id type nativeType name subscriptionId subscriptionExternalId subscriptionName resourceGroupId resourceGroupExternalId region cloudPlatform cloudProviderURL providerId status tags subscriptionTags } note serviceTicket { externalId name url } serviceTickets { externalId name url action { id type } } }
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> successfully generated new access token INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Terminated INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Setup for module <WizDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Started. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} WARNING InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Terminated 2INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Started INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 5 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Updating deduplication buffers content INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Terminated INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
Note that a Partial
Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial
reference.
(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
CloudConfiguration Service
All events of this service are ingested into the table cspm.wiz.issues.default
Issue service is based on the following GraphQL command:
query IssuesTable( $filterBy: IssueFilters $first: Int $after: String $orderBy: IssueOrder ) { issues( filterBy: $filterBy first: $first after: $after orderBy: $orderBy ) { nodes { ...IssueDetails } pageInfo { hasNextPage endCursor } totalCount informationalSeverityCount lowSeverityCount mediumSeverityCount highSeverityCount criticalSeverityCount uniqueEntityCount } } fragment IssueDetails on Issue { id control { id name query securitySubCategories { id title category { id name framework { id name } } } } createdAt updatedAt projects { id name slug businessUnit riskProfile { businessImpact } } status severity entity { id name type } entitySnapshot { id type nativeType name subscriptionId subscriptionExternalId subscriptionName resourceGroupId resourceGroupExternalId region cloudPlatform cloudProviderURL providerId status tags subscriptionTags } note serviceTicket { externalId name url } serviceTickets { externalId name url action { id type } } }
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in a organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> successfully generated new access token INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> The credentials provided in the configuration have required permissions to request issues from Wiz server INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Terminated INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Setup for module <WizDataPuller> has been successfully executed
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull
action is executed only one time before the first run of the Pull
action.
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Started. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> User has specified 2022-01-01 00:00:00 as the datetime. Historical polling will consider this datetime for creating the default values. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> No saved state found, initializing with state: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} WARNING InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Saved state loaded: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(1970, 1, 1, 0, 0), 'buffer_ids_with_duplication_risk': []} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> PrePull Terminated 2INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Starting data collection every 60 seconds INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Started INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Fetching for issues from 2022-01-01T00:00:00 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Total number of issues in this poll: 45 INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 5, 12, 19, 13, 20, 193191), 'buffer_ids_with_duplication_risk': ['09992ee4-1450-44fa-951c-d5fc4815473a']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 1; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 20. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO OutputProcess::SyslogSender(standard_senders,syslog_sender_0) -> syslog_sender_0 -> Created sender: {"client_name": "collector-4ac42f93cffaa59c-9dc9f67c9-cgm84", "url": "sidecar-service-default.integrations-factory-collectors:601", "object_id": "140446617222352"} INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 20 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 9, 0, 1, 927011), 'buffer_ids_with_duplication_risk': ['87e301c5-d3b7-4c2b-9495-9163772b3517', '7c95e45f-694e-4843-8aa7-d697a66fb14a', '5f3daede-c375-424f-9034-d9f423310b4a', '584ac078-87f2-45a5-b2eb-6e72e0594bd7', '5057cb24-ce5b-405d-bd5d-fd7b3ba70fc0', '22933fcb-ebb0-4a03-bb00-c1cba0b5abca', '1bed50e0-7825-41c9-a9de-8d32e0a35de8', '03a303c8-000c-4544-8f2c-65486a225e15']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Requesting Wiz API for issues INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> successfully retried issues from Wiz INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Removing the duplicate issues if present INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Flatten data is set to True. Flattening the data and adding 'devo_pulling_id' to events INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Delivering issues to the SDK INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> 5 issues delivered INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> State has been updated during pagination: {'historic_date_utc': datetime.datetime(2022, 1, 1, 0, 0), 'last_polled_timestamp': datetime.datetime(2022, 1, 1, 0, 0), 'ids_with_same_timestamp': [], 'buffer_timestamp_with_duplication_risk': datetime.datetime(2022, 6, 30, 13, 14, 40, 673424), 'buffer_ids_with_duplication_risk': ['4d819843-61ef-4e70-a2b6-5834a3f96403']}. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Updating deduplication buffers content INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797. INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Pull Terminated INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Data collection completed. Elapsed time: 1.334 seconds. Waiting for 58.666 second(s)
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
INFO InputProcess::WizDataPuller(wiz_data_puller,00011,issues,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1656602793.044179):Number of requests made: 3; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 45; Average of events per second: 33.797.
The value @devo_pulling_id
is injected in each event to group all events ingested by the same pull action. You can use it to get the exact events downloaded in that Pull
action in Devo’s search window.
Note that a Partial
Statistics Report will be displayed when pagination is required in order to pull all available events. Look for the report without the Partial
reference.
(Partial) Statistics for this pull cycle (@devo_pulling_id=1656602793.044179) so far: Number of requests made: 2; Number of events received: 45; Number of duplicated events filtered out: 0; Number of events generated and sent: 40.
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the
historical_date_utc
parameter to a different one.Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
Error type | Error ID | Error message | Cause | Solution |
---|---|---|---|---|
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when optional value | Edit the value of |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when the optional value | Edit the value of |
|
|
| This error is raised when the optional value | Edit the value of |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when | This is an internal issue. Contact with Devo Support team. |
|
|
| This error could be raised for two reasons:
| Solutions for both cases, respectively:
|
|
|
| This error could be raised for two reasons:
| Solutions for both cases, respectively:
|
|
|
| This error is raised when the required property | Add |
|
|
| This error is raised when | Edit the value of |
|
|
| This error is raised when the required property | Add |
|
|
| This error is raised when | Edit the value of |
|
|
| This error is raised when the required property | Add |
|
|
| This error is raised when | Edit the value of |
|
|
| This error is raised when the optional value | Edit the value of |
|
|
| This error is raised when the optional value | Make the value of |
|
|
| This error is raised when the optional value | Make the value of |
|
|
| This error is raised when a generic error occurs during the request to get the token to authenticate the collector in the API. | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when the credentials provided in | Add the correct |
|
|
| This error is raised when the credentials provided in | Add some credentials in |
|
|
| This error is raised when the credentials provided in | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when the credentials provided in | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when the credentials provided in | Check that the provided credentials have enough permission to retrieve issues from the API. |
|
|
| This error is raised when the credentials provided in | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when a generic error occurs during the request to get issues from the API. | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when the token being used to make requests to the API is not valid anymore. | Check if the credentials need to be renewed. Add some valid credentials in If credentials are still valid, contact with Devo Support team. |
|
|
| This error is raised when the token being used to make requests to the API is valid, but it lost permissions to get issues from API. | Check what happened with credential permissions. Add some credentials in If credentials are valide and still have permissions, contact with Devo Support team. |
|
|
| This error is raised when the token being used to make requests to the API is valid, but cannot find the endpoints to get issues. | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when the token being used to make requests to the API is valid, but there has been an error on Wiz's API. | The error is on Wiz’s side. Wiz can be contacted for more info. It should work again when the incident at Wiz is solved. |
|
|
| This error is raised when the token being used to make requests to the API is valid, but there has been an unexpected return from the API. | This is an internal issue. Contact with Devo Support team. |
|
|
| This error is raised when the token being used to make requests to the API is valid, but we are constantly receiving a | Check throttle limitations on Wiz API and change the value of |
Collector operations
This section is intended to explain how to proceed with specific operations of this collector.
Initialization
The initialization module is in charge of setup and running the input (pulling logic) and output (delivering logic) services and validating the given configuration.
A successful run has the following output messages for the initializer module:
INFO MainProcess::MainThread -> (CollectorMultiprocessingQueue) standard_queue_multiprocessing -> max_size_in_messages: 1000, max_size_in_mb: 1024, max_wrap_size_in_items: 100 INFO MainProcess::MainThread -> [OUTPUT] OutputMultiprocessingController::__init__ Configuration -> {'devo_1': {'type': 'devo_platform', 'config': {'address': 'collector-eu.devo.io', 'port': 443, ...}}} INFO MainProcess::MainThread -> OutputProcess - Starting thread (executing_period=300s) INFO MainProcess::MainThread -> InputProcess - Starting thread (executing_period=300s) INFO OutputProcess::MainThread -> Process started INFO InputProcess::MainThread -> Process Started INFO InputProcess::MainThread -> InitVariables Started INFO InputProcess::MainThread -> Validating variables in collector definitions Started INFO InputProcess::MainThread -> Flatten data is not provided in the config.yaml. Considering the flatten data from collector definitions INFO InputProcess::MainThread -> Validating collector Variables is terminated INFO InputProcess::MainThread -> Initialization of api_base_url has started. INFO InputProcess::MainThread -> api_base_url has been initialized INFO InputProcess::MainThread -> Initialization of credentials has started. INFO InputProcess::MainThread -> credentials have been initialized. INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io INFO InputProcess::MainThread -> InitVariables Terminated INFO InputProcess::MainThread -> InputThread(wiz_data_puller,111) - Starting thread (execution_period=120s) INFO InputProcess::MainThread -> ServiceThread(wiz_data_puller,111,issues,predefined) - Starting thread (execution_period=120s) INFO InputProcess::MainThread -> WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Starting thread INFO InputProcess::MainThread -> WizDataPuller(wiz_data_puller,111,issues,predefined) - Starting thread WARNING InputProcess::WizDataPuller(wiz_data_puller,111,issues,predefined) -> Waiting until setup will be executed INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Puller Setup Started INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> This is the first run of collector. Generating the access token INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Getting the auth token url based on provided api_base_url INFO InputProcess::WizDataPullerSetup(wiz_collector,wiz_data_puller#111,issues#predefined) -> Using default Authentication Domain auth.wiz.io for fetching Access Token INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io INFO OutputProcess::MainThread -> [INTERNAL LOGIC] DevoSender::_validate_kwargs_for_method__init__ -> The <address> does not appear to be an IP address and cannot be verified: collector-eu.devo.io INFO OutputProcess::MainThread -> DevoSender(standard_senders,devo_sender_0) -> Starting thread INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(standard_senders,devo_1) -> Starting thread (every 300 seconds) INFO OutputProcess::MainThread -> DevoSenderManager(standard_senders,manager,devo_1) -> Starting thread INFO OutputProcess::MainThread -> DevoSender(lookup_senders,devo_sender_0) -> Starting thread INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(lookup_senders,devo_1) -> Starting thread (every 300 seconds) INFO OutputProcess::MainThread -> DevoSenderManager(lookup_senders,manager,devo_1) -> Starting thread INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_1) -> Starting thread (every 300 seconds) INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_1) -> Starting thread INFO InputProcess::MainThread -> [GC] global: 36.7% -> 36.7%, process: RSS(26.93MiB -> 27.97MiB), VMS(334.43MiB -> 334.67MiB) INFO OutputProcess::MainThread -> [GC] global: 36.7% -> 36.3%, process: RSS(26.68MiB -> 28.61MiB), VMS(910.71MiB -> 910.71MiB) INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"group_name": "internal_senders", "instance_name": "devo_sender_0", "url": "collector-eu.devo.io:443", ...}
Events delivery and Devo ingestion
The event delivery module is in charge of receiving the events from the internal queues where all events are injected by the pullers and delivering them using the selected compatible delivery method.
A successful run has the following output messages for the initializer module:
INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0 INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {} INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Sender: SyslogSender(standard_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} INFO OutputProcess::SyslogSenderManagerMonitor(standard_senders,sidecar_0) -> Standard - Total number of messages sent: 44, messages sent since "2022-06-28 10:39:22.511671+00:00": 44 (elapsed 0.007 seconds) INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Number of available senders: 1, sender manager internal queue size: 0 INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> enqueued_elapsed_times_in_seconds_stats: {} INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Sender: SyslogSender(internal_senders,syslog_sender_0), status: {"internal_queue_size": 0, "is_connection_open": True} INFO OutputProcess::SyslogSenderManagerMonitor(internal_senders,sidecar_0) -> Internal - Total number of messages sent: 1, messages sent since "2022-06-28 10:39:22.516313+00:00": 1 (elapsed 0.019 seconds)
By default, these information traces will be displayed every 10 minutes.
Sender services
The Integrations Factory Collector SDK has 3 different senders services depending on the event type to delivery (internal
, standard
, and lookup
). This collector uses the following Sender Services:
Sender services | Description |
---|---|
| In charge of delivering internal metrics to Devo such as logging traces or metrics. |
| In charge of delivering pulled events to Devo. |
Sender statistics
Each service displays its own performance statistics that allow checking how many events have been delivered to Devo by type:
Logging trace | Description |
---|---|
| Displays the number of concurrent senders available for the given Sender Service. |
| Displays the items available in the internal sender queue. This value helps detect bottlenecks and needs to increase the performance of data delivery to Devo. This last can be made by increasing the concurrent senders. |
| Displayes the number of events from the last time and following the given example, the following conclusions can be obtained:
By default these traces will be shown every 10 minutes. |
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the
global
value.All metrics (Global, RSS, VMS) include the value before freeing and after
previous -> after freeing memory
INFO InputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(34.50MiB -> 34.08MiB), VMS(410.52MiB -> 410.02MiB) INFO OutputProcess::MainThread -> [GC] global: 20.4% -> 20.4%, process: RSS(28.41MiB -> 28.41MiB), VMS(705.28MiB -> 705.28MiB)
Differences between RSS
and VMS
memory usage:
RSS
is the Resident Set Size, which is the actual physical memory the process is usingVMS
is the Virtual Memory Size which is the virtual memory that process is using
Sometimes it is necessary to activate the debug mode of the collector's logging. This debug mode increases the verbosity of the log and allows you to print execution traces that are very helpful in resolving incidents or detecting bottlenecks in heavy download processes.
To enable this option you just need to edit the configuration file and change the debug_status parameter from false to true and restart the collector.
To disable this option, you just need to update the configuration file and change the debug_status parameter from true to false and restart the collector.
For more information, visit the configuration and parameterization section corresponding to the chosen deployment mode.
Change log
Release | Released on | Release type | Details | Recommendations |
---|---|---|---|---|
| FEATUREIMPROVEMENTS | New Features
Improvements
|
| |
| FEATUREIMPROVEMENTS | New Features
Improvements
|
| |
| BUG FIXIMPROVEMENTS | Improvements:
Bug Fix:
|
| |
| IMPROVEMENTS | Improvements:
|
| |
| BUG FIX | Bug fixes:
|
| |
| FEATURE | New features:
|
| |
| FEATURE | New features:
|
|
- No labels