Netskope API V2 collector

[ 1 Overview ] [ 2 Netskope API V2 collector migration guide (from 1.x.x to 2.x.x) ] [ 3 Devo collector features ] [ 4 Data sources ] [ 5 API Limits, Delays, Known Issues ] [ 6 Accepted authentication methods ] [ 7 Minimum configuration required for basic pulling ] [ 8 Vendor setup ] [ 9 Rate limiting ] [ 10 Run the collector ] [ 11 Collector service details ] [ 12 Change log ]

Overview

Netskope Cloud Access Security Broker (CASB) is a security solution designed to provide visibility, control, and protection for data and applications in cloud environments. CASBs address the security challenges posed by the increasing adoption of cloud services, offering a layer of security between cloud service users and cloud applications to enforce security policies.

Netskope CASB Collector requests logs from Netskope APIs and sends them to Devo.

Netskope API V2 collector migration guide (from 1.x.x to 2.x.x)

If you need to migrate an old collector version to a more recent one, please check the migration process in this article.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`not allowed`
Running environments	`collector server` `on-premise`
Populated Devo events	`table`

Data sources

The data is collected using a Devo collector that can be run on the Devo collector server or standalone in a Docker container. The data is sent and stored in the Devo platform in these tables:

Data source	Description	API endpoint	Collector service name	Devo table	Available from release

Data source	Description	API endpoint	Collector service name	Devo table	Available from release
Event alert	Get all the Alert event type.	/api/v2/events/dataexport/events/alert	`event_alert`	`casb.netskope.alert`	`v1.0.0`
Event page	Get all the page event type.	/api/v2/events/dataexport/events/page	`event_page`	`casb.netskope.page`	`v1.0.0`
Event application	Get all the application event type.	/api/v2/events/dataexport/events/application	`event_application`	`casb.netskope.application`	`v1.0.0`
Event incident	Get all the incident event type.	/api/v2/events/dataexport/events/incident	`event_incident`	`casb.netskope.incident`	`v1.0.0`
Event audit	Get all the audit event type.	/api/v2/events/dataexport/events/audit	`event_audit`	`casb.netskope.audit`	`v1.0.0`
Event infrastructure	Get all the infrastructure event type.	/api/v2/events/dataexport/events/infrastructure	`event_infrastructure`	`casb.netskope.infrastructure`	`v1.0.0`
Event network	Get all the network event type.	/api/v2/events/dataexport/events/network	`event_network`	`casb.netskope.network`	`v1.0.0`
Alert dlp	Get all the dlp Alert type.	/api/v2/events/dataexport/alerts/dlp	`alert_dlp`	`casb.netskope.dlp`	`v1.0.0`
Alert watchlist	Get all the watchlist Alert type.	/api/v2/events/dataexport/alerts/watchlist	`alert_watchlist`	`casb.netskope.watchlist`	`v1.0.0`
Alert ctep	Get all the ctep Alert type.	/api/v2/events/dataexport/alerts/ctep	`alert_ctep`	`casb.netskope.ctep`	`v1.0.0`
Alert compromisedcredential	Get all the compromisedcredential Alert type.	/api/v2/events/dataexport/alerts/compromisedcredential	`alert_compromisedcredential`	`casb.netskope.compromisedcredential`	`v1.0.0`
Alert malsite	Get all the malsite Alert type.	/api/v2/events/dataexport/alerts/malsite	`alert_malsite`	`casb.netskope.malsite`	`v1.0.0`
Alert malware	Get all the malware Alert type.	/api/v2/events/dataexport/alerts/malware	`alert_malware`	`casb.netskope.malware`	`v1.0.0`
Alert policy	Get all the policy Alert type.	/api/v2/events/dataexport/alerts/policy	`alert_policy`	`casb.netskope.policy`	`v1.0.0`
Alert remediation	Get all the remediation Alert type.	/api/v2/events/dataexport/alerts/remediation	`alert_remediation`	`casb.netskope.remediation`	`v1.0.0`
Alert quarantine	Get all the quarantine Alert type.	/api/v2/events/dataexport/alerts/quarantine	`alert_quarantine`	`casb.netskope.quarantine`	`v1.0.0`
Alert securityassessment	Get all the securityassessment Alert type.	/api/v2/events/dataexport/alerts/securityassessment	`alert_securityassessment`	`casb.netskope.securityassessment`	`v1.0.0`
Alert uba	Get all the uba Alert type.	/api/v2/events/dataexport/alerts/uba	`alert_uba`	`casb.netskope.uba`	`v1.0.0`

More information about the API calls can be found here.

For more information on how the events are parsed, visit our page.

API Limits, Delays, Known Issues

We occasionally encounter a 409 "Concurrency conflict" error, indicating the request cannot be processed at this point of time(Implemented wait time according to API document but still getting this error). This is a known limitation, and handled this. It won't break the collector.
Events may appear to be delayed in Devo due to time zone differences. This is not an actual delay, because timestamps are based on the source data's time zone, not UTC, leading to perceived discrepancies.

Accepted authentication methods

Authentication method	api_token

Authentication method	api_token
Auth token	REQUIRED

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details

Setting	Details
api_token	The Api Token for netskope API.

Vendor setup

Netskope API collector works over the API to retrieve the data, so a token is required to get the data via API. Follow the steps here to get an API token.

Rate limiting

Rate-limiting must be factored in when using the Netskope REST APIs. A standard 429 Too Many Requests error will be returned if an excessive usage level is reached. To avoid this error, limit your REST API calls. The global rate limit can be checked at the top of the page.

In this example (4 req/s), limit the API calls to no more than 20 requests every 5 seconds. Four requests are processed in the first second, while 16 are queued and processed over the next four seconds.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector service details

Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in an organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2025-01-28T10:52:34.101    INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2025-01-28T10:52:34.101    INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2025-01-28T10:52:34.101    INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2025-01-28T10:52:34.102    INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_2) -> Starting thread (every 300 seconds)
2025-01-28T10:52:34.102    INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> Starting thread
2025-01-28T10:52:34.103    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2025-01-28T10:52:34.103    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2025-01-28T10:52:34.104    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2025-01-28T10:52:34.104    INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2025-01-28T10:52:34.105    INFO OutputProcess::MainThread -> OutputMetricsThread -> Started thread for updating metrics values (update_period=10.0)
2025-01-28T10:52:34.106    INFO InputProcess::MainThread -> CollectorPuller(netskope_v2api#45678,event_page#predefined) Finalizing the execution of init_variables()
2025-01-28T10:52:34.110    INFO InputProcess::MainThread -> InputThread(netskope_v2api,45678) - Starting thread (execution_period=60s)
2025-01-28T10:52:34.112    INFO InputProcess::MainThread -> ServiceThread(netskope_v2api,45678,event_page,predefined) - Starting thread (execution_period=60s)
2025-01-28T10:52:34.113    INFO InputProcess::MainThread -> CollectorPullerSetup(netskope_v2api#45678,event_page#predefined) -> Starting thread
2025-01-28T10:52:34.113    INFO InputProcess::MainThread -> CollectorPuller(netskope_v2api#45678,event_page#predefined) - Starting thread
2025-01-28T10:52:34.118 WARNING InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Waiting until setup will be executed
2025-01-28T10:52:34.120    INFO InputProcess::MainThread -> InputMetricsThread -> Started thread for updating metrics values (update_period=10.0)
2025-01-28T10:52:34.123    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "vendor_requests" created: "Number of requests received from the vendor API", unit: "requests"
2025-01-28T10:52:34.159 WARNING MainProcess::CollectorThread -> There is a process that now it doesn't exists (pid=91533)
2025-01-28T10:52:34.171    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2025-01-28T10:52:34.171    INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.07
2025-01-28T10:52:34.171    INFO OutputProcess::MainThread -> [GC] global: 55.6% -> 55.7%, process: RSS(62.14MiB -> 62.76MiB), VMS(1.07GiB -> 1.07GiB)
2025-01-28T10:52:34.172    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_counter" created: "Number of messages sent to the defined output", unit: "1"
2025-01-28T10:52:34.173    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_sent_bytes" created: "Number of bytes sent to the defined output", unit: "1"
2025-01-28T10:52:34.174    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2025-01-28T10:52:34.174    INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.07
2025-01-28T10:52:34.176    INFO InputProcess::MainThread -> [GC] global: 55.7% -> 55.7%, process: RSS(62.14MiB -> 62.64MiB), VMS(521.70MiB -> 521.70MiB)
2025-01-28T10:52:34.177    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2025-01-28T10:52:34.177    INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.07
2025-01-28T10:52:34.177    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_received" created: "Number of messages received from the vendor API", unit: "1"
2025-01-28T10:52:34.179    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_removed" created: "Number of messages removed by the collector", unit: "1"
2025-01-28T10:52:34.180    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_incoming_filtered" created: "Number of messages filtered by the collector", unit: "1"
2025-01-28T10:52:34.181    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_counter" created: "Number of messages enqueued", unit: "1"
2025-01-28T10:52:34.183    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_standard_bytes" created: "Number of bytes enqueued", unit: "1"
2025-01-28T10:52:34.184    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_counter" created: "Number of messages enqueued", unit: "1"
2025-01-28T10:52:34.185    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_lookup_bytes" created: "Number of messages enqueued", unit: "1"
2025-01-28T10:52:34.186    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_counter" created: "Number of messages enqueued in the queue", unit: "1"
2025-01-28T10:52:34.186    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Counter "msg_enqueued_internal_bytes" created: "Number of messages enqueued in the queue", unit: "1"
2025-01-28T10:52:34.187    INFO MainProcess::MetricsConsumerThread -> OpenTelemetryServer -> [METRIC] Gauge "module_global_status" created: "Global status of current module", unit: "1"
2025-01-28T10:52:34.558    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/pulkit/devo/collectors/devo-collector-netskope-api-v2/certs/chain.crt", "cert_path": "/home/pulkit/devo/collectors/devo-collector-netskope-api-v2/certs/int-if-integrations-india.crt", "key_path": "/home/pulkit/devo/collectors/devo-collector-netskope-api-v2/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-APAC-0049", session_id: "132784344291840"
2025-01-28T10:52:34.559    INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2025-01-28T10:52:35.553    INFO InputProcess::CollectorPullerSetup(netskope_v2api#45678,event_page#predefined) -> Setup for module <CollectorPuller> has been successfully executed
2025-01-28T10:52:36.123    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> CollectorPuller(netskope_v2api#45678,event_page#predefined) Starting the execution of pre_pull()
2025-01-28T10:52:36.125    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Reading persisted data
2025-01-28T10:52:36.127    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Data retrieved from the persistence: None
2025-01-28T10:52:36.128 WARNING InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Persistence will be overridden due to the retrieved state is empty
2025-01-28T10:52:36.129    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Running the persistence upgrade steps
2025-01-28T10:52:36.129    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Running the persistence corrections steps
2025-01-28T10:52:36.129    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Running the persistence corrections steps
2025-01-28T10:52:36.130 WARNING InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Some changes have been detected and the persistence needs to be updated. Previous content: None. New content: {'@persistence_version': 1, 'initial_start_time_in_epoch': 1738041754, 'last_time_in_epoch': 1738041754, 'last_ids': []}
2025-01-28T10:52:36.131    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Updating the persistence
2025-01-28T10:52:36.133 WARNING InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Persistence has been updated successfully
2025-01-28T10:52:36.133    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> CollectorPuller(netskope_v2api#45678,event_page#predefined) Finalizing the execution of pre_pull()
2025-01-28T10:52:36.133    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Starting data collection every 60 seconds

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2025-01-28T10:52:36.134    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Pull Started
2025-01-28T10:52:37.786    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Updating the persistence
2025-01-28T10:53:07.818    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1738041756123):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2025-01-28T10:53:07.820    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1738041756123):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2025-01-28T10:53:07.821    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> The data is up to date!

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2025-01-28T10:55:11.251    INFO InputProcess::CollectorPuller(netskope_v2api#45678,event_page#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1738041879396):Number of requests made: 1; Number of events received: 102; Number of duplicated events filtered out: 0; Number of events generated and sent: 102; Average of events per second: 3.202.

Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

Edit the configuration file.
Change the value of the initial_start_time_in_utc_value parameter to a different one.
Save the changes.
Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

[ Troubleshooting ]

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

[ Common Logic ]

Error Type	Error Id	Error Message	Cause	Solution

Error Type	Error Id	Error Message	Cause	Solution
InitVariablesError	1	Date {start_time_in_utc} is either in the future or older than 7 days.	The date in config is not in valid time period which is last 7 days.	Ensure the datetime is less than current time and not older than 7 days.
ApiError	400	Some error occurred while retrieving events from Netskope.	Wrong credentials or something else from API side.	Check the credentials and ensure that the collector has the necessary permissions to access the Netskope API.

Check Memory Usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

Differences between RSS and VMS memory usage:

RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using

Change log

Release	Released on	Release type	Details	Recommendations

Release	Released on	Release type	Details	Recommendations
`2.0.1`	Jan 30, 2025	IMPROVEMENTS	Improvements Added gitlab pipeline.	`Recommended Version`
`2.0.0`	Jan 23, 2025	IMPROVEMENTS Changed Fixed	Improvements New, refactored version. See Migration guide Fixed Fixed Duplication and Delay issue. Fixed 409 Error by adding wait time. Changed Updated the DCSDK from 1.11.1 to 1.13.1 Added new sender for relay in house + TLS Added persistence functionality for gzip sending buffer Added Automatic activation of gzip sending Improved behaviour when persistence fails Upgraded DevoSDK dependency Fixed console log encoding Restructured python classes Improved behaviour with non-utf8 characters Decreased default size value for internal queues (Redis limitation, from 1GiB to 256MiB) New persistence format/structure (compression in some cases) Removed dmesg execution (It was invalid for docker execution) Applied changes to make DCSDK compatible with MacOS Upgrade DevoSDK dependency to version v5.4.0 Change internal queue management for protecting against OOMK Extracted ModuleThread structure from PullerAbstract Improve Controlled stop when both processes fails to instantiate Improve Controlled stop when InputProcess is killed Bug related to lost of collector_name , collector_id and job_id Bug related queues and ValueError (edited) Change internal queue management for protecting against OOMK Extracted ModuleThread structure from PullerAbstract Improve Controlled stop when both processes fails to instantiate Improve Controlled stop when InputProcess is killed Fixed error related a ValueError exception not well controlled Fixed error related with loss of some values in internal messages Upgraded dcsdk-docker-base-image to 1.3.1	`Recommended Version`
`v1.1.0`	May 30, 2024	IMPROVEMENTS	Improvements: Updated DC SDK to `v1.11.1` Updated Docker image base to version `v1.2.0` in Dockerfile	`Upgrade`
`v1.0.1`	Sep 25, 2023	IMPROVEMENTS	Improvements: Update default configuration values to avoid rate-limiting	`Upgrade`
`v1.0.0`	Aug 29, 2023	FEATURE	New features: Updating to newest SDK 1.6.2 to 1.9.2 Upgrade internal dependencies Store lookup instances into DevoSender to avoid the creation of new instances for the same lookup Ensure service_config is a dict into templates Ensure special characters are properly sent to the platform Changed log level to some messages from info to debug Changed some wrong log messages Upgraded some internal dependencies Changed queue passed to setup instance constructor Ability to validate collector setup and exit without pulling any data Ability to store in the persistence the messages that couldn't be sent after the collector stopped Ability to send messages from the persistence when the collector starts and before the puller begins working Ensure the special characters are properly sent to the platform Added a lock to enhance the sender object Added new class attrs to the setstate and getstate queue methods Fix sending attribute value to the setstate and getstate queue methods Added log traces when queues are full and have to wait Added log traces of queue time waiting every minute in debug mode Added method to calculate queue size in bytes Block incoming events in queues when there is no space left Send telemetry events to the Devo platform Upgraded internal Python dependency Redis to v4.5.4 Upgraded internal Python dependency DevoSDK to v5.1.3 Fixed obfuscation not working when messages are sent from templates New method to figure out if a puller thread is stopping Upgraded internal Python dependency DevoSDK to v5.0.6 Improved logging on messages/bytes sent to the Devo platform Fixed wrong byte size calculation for queues	`Upgrade`

Devo latest