Document toolboxDocument toolbox

Okta Resources collector

Configuration requirements

To run this collector, there are some configurations detailed below that you need to consider.

Configuration

Details

Configuration

Details

Getting Okta credentials

You will need to create an api_token and get the okta_url to run this collector.

More information

Refer to the Vendor setup section to know more about these configurations.

Overview

The Okta Resources API is used for gaining insights into the content management of activities from your organization or company. Okta Resources APIs generate system logs and other events in real time.

Okta collector migration guide (from 1.x.x to 2.x.x)

If you need to migrate an old collector version to a more recent one, please check the migration process in this article.

Devo Collector features

Feature

Details

Feature

Details

Allow parallel downloading (multipod)

Not allowed

Running Environments

Collector Server

On Premise

Populated Devo events

table

Flattening preprocessing

no

Data source description

You can use the Okta collector to send this information to your Devo domain. Once the gathered information arrives at Devo it will be categorized in different tables in your domain, as you can check in the following table.

Okta services

Listed in the table below are some service names, details, and how the Devo platform treats the data.

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

Data source

Description

API endpoint

Collector service name

Devo table

Available from release

System Logs

System Log records system events related to your organization in order to provide an audit trail that can be used to understand platform activity and to diagnose problems. Often the terms "event" and "log event" are used interchangeably. In the context of this API, an "event" is an occurrence of interest within the system and "log" or "log event" is the recorded fact.

/v1/logs

logs

auth.okta.system

v1.0.0

Apps

Application API provides operations to manage applications and/or assignments to users or groups for your organization.

/v1/apps

apps

auth.okta.apps

v1.0.0

Client Application

The Dynamic Client Registration API provides operations to register and manage client applications to be used with Okta's OAuth 2.0 and OpenID Connect endpoints.

/v1/clients

clients

auth.okta.clients

v1.0.0

Groups

Groups API provides operations to manage Okta groups and their user members for your organization.

/v1/groups

groups

auth.okta.groups

v1.0.0

IDPS

Identity Providers API provides operations to manage federations with external Identity Providers (IDP). For example, your app can support logging in with credentials from Facebook, Google, LinkedIn, Microsoft, an enterprise IdP using SAML 2.0, or an IdP using the OpenID Connect (OIDC) protocol.

/v1/idps

idps

auth.okta.idps

v1.0.0

Users

User API provides operations to manage users in your organization.

/v1/users

users

auth.okta.users

v1.0.0

Zones

Zones API provides operations to manage zones in your organization. Zones may be used to guide policy decisions.

/v1/zones

zones

auth.okta.zones

v1.0.0

The System Log API will eventually replace the Events API. It contains much more structured data.

For more references about Okta Resources API, visit the Okta API Reference.

Accepted authentication methods

Authentication method

okta_url

api_token

Authentication method

okta_url

api_token

API-token pair

REQUIRED

REQUIRED

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting

Details

Setting

Details

okta_url

The okta url for okta API.

api_token

The api token for okta API.

Vendor Setup

Getting Okta credentials

  1. Visit Developer Okta to create an api_token and get the okta_url.

  2. Log in with your company credentials (or sign up for a free developer account).

  3. Click Dashboard and save the okta_url that is displayed in the top right corner (it will be used later in the config file).


  4. On the top menu, go to API → Tokens.


  5. Click on Create Token and enter a name for your token in the window that appears, which will be used for tracking API calls. Click Create Token.


  6. Copy your token and click OK, got it. Note that the token will be only displayed here, so don't forget to copy it. Save it as api_token (it will be used later in the config file).


Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector service details

Verify data collection

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component

Description

Component

Description

Setup

The setup module is in charge of authenticating the service and managing the token expiration when needed.

Puller

The setup module is in charge of pulling the data in an organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2024-12-04T17:15:59.197 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread 2024-12-04T17:15:59.197 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_2) -> Starting thread (every 300 seconds) 2024-12-04T17:15:59.197 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> Starting thread 2024-12-04T17:15:59.198 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-04T17:15:59.198 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system 2024-12-04T17:15:59.201 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-04T17:15:59.201 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00 2024-12-04T17:15:59.223 INFO InputProcess::MainThread -> [GC] global: 60.6% -> 60.7%, process: RSS(41.81MiB -> 41.81MiB), VMS(424.86MiB -> 424.86MiB) 2024-12-04T17:15:59.225 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-04T17:15:59.225 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03 2024-12-04T17:15:59.225 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-04T17:15:59.225 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03 2024-12-04T17:15:59.226 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-04T17:15:59.226 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03 2024-12-04T17:15:59.227 INFO OutputProcess::MainThread -> [GC] global: 60.6% -> 60.7%, process: RSS(41.55MiB -> 41.92MiB), VMS(929.13MiB -> 929.38MiB) 2024-12-04T17:15:59.227 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-04T17:15:59.227 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03 2024-12-04T17:15:59.228 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-04T17:15:59.228 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03 2024-12-04T17:15:59.744 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/pulkit/devo/collectors/devo-collector-okta/certs/chain.crt", "cert_path": "/home/pulkit/devo/collectors/devo-collector-okta/certs/int-if-integrations-india.crt", "key_path": "/home/pulkit/devo/collectors/devo-collector-okta/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-APAC-0049", session_id: "124538478558176" 2024-12-04T17:15:59.744 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system 2024-12-04T17:16:00.360 INFO InputProcess::OktaResourcesPullerSetup(okta_collector,okta#65294,logs#predefined) -> Setup for module <OktaResourcesPuller> has been successfully executed 2024-12-04T17:16:01.224 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> OktaResourcesPuller(okta#65294,logs#predefined) Starting the execution of pre_pull() 2024-12-04T17:16:01.225 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Reading persisted data 2024-12-04T17:16:01.227 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Data retrieved from the persistence: None 2024-12-04T17:16:01.228 WARNING InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Persistence will be overridden due to the retrieved state is empty 2024-12-04T17:16:01.229 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Running the persistence upgrade steps 2024-12-04T17:16:01.230 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Running the persistence corrections steps 2024-12-04T17:16:01.230 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Running the persistence corrections steps 2024-12-04T17:16:01.231 WARNING InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Some changes have been detected and the persistence needs to be updated. Previous content: None. New content: {'@persistence_version': 1, 'next_link': '', 'initial_start_time_in_utc': '2024-12-04T11:45:01Z', 'last_timestamp': '2024-12-04T11:45:01Z', 'uuid_list': []} 2024-12-04T17:16:01.233 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Updating the persistence 2024-12-04T17:16:01.235 WARNING InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Persistence has been updated successfully 2024-12-04T17:16:01.235 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> OktaResourcesPuller(okta#65294,logs#predefined) Finalizing the execution of pre_pull() 2024-12-04T17:16:01.236 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Starting data collection every 60 seconds 2024-12-04T17:16:01.236 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Pull Started 2024-12-04T17:16:02.384 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1733312761224):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2024-12-04T17:16:01.236 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Pull Started 2024-12-04T17:16:02.384 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1733312761224):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. 2024-12-04T17:16:02.384 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1733312761224):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000. 2024-12-04T17:16:02.385 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> The data is up to date!

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2024-12-04T17:17:55.904 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1733312874612):Number of requests made: 1; Number of events received: 255; Number of duplicated events filtered out: 0; Number of events generated and sent: 255; Average of events per second: 198.357.

Restart the persistence

This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:

  1. Edit the configuration file.

  2. Change the value of the start parameter to a different one.

  3. Save the changes.

  4. Restart the collector.

The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.

Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.

[ Troubleshooting ]

This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.

[ Common Logic ]

Error Type

Error Id

Error Message

Cause

Solution

Error Type

Error Id

Error Message

Cause

Solution

InitVariablesError

1

initial_start_time_in_utc is not set as per the datetime_format : {datetime_format}

The date in config is not as per required format

Ensure the date format is correct.

InitVariablesError

2

initial_start_time_in_utc is in the future

The date in config is not in the past period

Ensure the date period is correct.

SetupError

100

HTTP Error occurred while retrieving events from okta server

Okta API call is failing

Check the credentials and ensure that the collector has the necessary permissions to access the Okta API.

SetupError

101

Connecting Error occurred while retrieving events from okta server

Okta API call is failing

Check the credentials and ensure that the collector has the necessary permissions to access the Okta API.

PullError

300

HTTP Error occurred while fetching siem event from okta server

Okta API call is failing

Check the credentials and ensure that the collector has the necessary permissions to access the Okta API.

PullError

301

HTTP Error occurred while fetching siem event from okta server

Okta API call is failing

Check the credentials and ensure that the collector has the necessary permissions to access the Okta API.

PullError

302

Some error occurred while retrieving events from Okta server.

Okta API call is failing

Check the credentials and ensure that the collector has the necessary permissions to access the Okta API.

Check Memory Usage

To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.

  • The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.

  • The global pressure of the available memory is displayed in the global value.

  • All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory

Differences between RSS and VMS memory usage:

  • RSS is the Resident Set Size, which is the actual physical memory the process is using

  • VMS is the Virtual Memory Size which is the virtual memory that process is using

Activeboards

There are a number of predefined dashboards that can be downloaded in your Devo domain.

  1. Create a new Devo Activeboard in your domain.

  2. In the Edit mode click on the ellipsis button and select Edit raw configuration.

  3. Open the downloaded file, select all the text, and copy it to the clipboard.

  4. Paste the content of the file in the raw editor. Make sure you replace completely the existing configuration.

  5. Click on Save the changes. The dashboard shows up.

Rate limits

The number of API requests for an organization is limited for all APIs in order to protect the service for all users. The number of Okta-generated emails that can be sent also has rate limits.

Okta has two types of API rate limits:

  • Org-wide rate limits that vary by API endpoint. These limits are applied on a per-minute or per-second basis, and some are also applied on a per-user basis. For example, if your org sends a request to list applications more than one hundred times in a minute, the org-wide rate limit is exceeded. These limits protect against denial-of-service attacks and help ensure that adequate resources are available for all customers.

  • Concurrent rate limits on the number of simultaneous transactions. For example, if you sent 77 very long-lasting requests to any API endpoint simultaneously, you might exceed the concurrent rate limit.

Okta has one type of email rate limit:

  • Okta-Generated Email Message Rate Limits that vary by email type. Okta enforces rate limits on the number of Okta-generated email messages that are sent to customers and customer users. For example, if the number of emails sent to a given user exceeds the per-minute limit for a given email type, subsequent emails of that type are dropped for that user until that minute elapses.

Rate limits may be changed to protect customers. We provide advance warning of changes when possible.

Check the following web pages for more information on Okta rate limits:

Change log

Release

Released on

Release type

Details

Recommendations

Release

Released on

Release type

Details

Recommendations

v2.0.0

Dec 6, 2024

IMPROVEMENT

Improvements

  • Upgraded the DCSDK from 1.12.4 to 1.13.1

    • Change internal queue management for protecting against OOMK

    • Extracted ModuleThread structure from PullerAbstract

    • Improve Controlled stop when both processes fails to instantiate

    • Improve Controlled stop when InputProcess is killed

    • Fixed error related a ValueError exception not well controlled

  • Upgraded Docker image to 1.3.1

  • Refactor the code using template1

Recommended Version

v1.9.0

Aug 19, 2024

IMPROVEMENT

Improvements

  • Upgrade DCSDK to 1.12.4.

  • Upgrade the Docker Base image to 1.3.0.

Recommended Version

v1.8.1

Mar 18, 2024

IMPROVEMENT

Improvements:

  • Update DCSDK to version 1.11.1

Upgrade

v1.8.0

Sep 8, 2023

NEW FEATURE

New features:

Added an override_tag option in config.yaml. If set, it will use this tag instead of the one defined in collector_definitions.yaml

Upgrade

v1.7.0

Jul 3, 2023

IMPROVEMENT

Improvements:

  • Updated default rate limits for the following services:

    • logs: 20 requests per minute.

    • users: 100 requests per minute.

    • groups: 100 requests per minute.

    • apps: 20 requests per minute.

    • idps: 100 requests per minute.

    • zones: 100 requests per minute.

    • cumulative: 980 requests per minute.

  • Update DCSDK from 1.7.2 to 1.8.0:

    • Ability to validate collector setup and exit without pulling any data.

    • Ability to store in the persistence the messages that couldn't be sent after the collector stopped.

    • Ability to send messages from the persistence when the collector starts and before the puller begins working.

    • Ensure special characters are properly sent to the platform.

Upgrade

v1.6.0

Apr 25, 2023

IMPROVEMENT

Improvements:

  • Upgrade the Devo Collector SDK from v1.4.3 to v1.7.2.

  • Improve the way the duplicates are handled.

  • Some general refactoring.

Upgrade

v1.5.0

Nov 3, 2022

IMPROVEMENT

Improvements:

  • Upgraded underlay IFC SDK from v1.4.0 to v1.4.3.

    • Added:

      • New "templates" functionality.

      • New controlled stopping condition when any input thread fatally fails.

      • Added log traces for knowing the execution environment status (debug mode).

    • Changed:

      • Improved log trace details when runtime exceptions happen.

      • Refactored source code structure.

      • Fixes in the current puller template version.

      • The Docker container exits with the proper error code.

Upgrade

v1.4.3

Jul 29, 2022

IMPROVEMENT

Improvements:

  • Upgraded underlay IFC SDK v1.3.0 to v1.4.0.

    • Updated the underlying DevoSDK package to v3.6.4 and dependencies, this upgrade increases the resilience of the collector when the connection with Devo or the Syslog server is lost. The collector is able to reconnect in some scenarios without running the self-kill feature.

    • Support for stopping the collector when a GRACEFULL_SHUTDOWN system signal is received.

    • Re-enabled the logging to devo.collector.out for Input threads.

    • Improved self-kill functionality behavior.

    • Added more details in log traces.

    • Added log traces for knowing system memory usage.

Upgrade