Okta Resources collector
Configuration requirements
To run this collector, there are some configurations detailed below that you need to consider.
Configuration | Details |
---|---|
Getting Okta credentials | You will need to create an api_token and get the okta_url to run this collector. |
More information
Refer to the Vendor setup section to know more about these configurations.
Overview
The Okta Resources API is used for gaining insights into the content management of activities from your organization or company. Okta Resources APIs generate system logs and other events in real time.
Okta collector migration guide (from 1.x.x to 2.x.x)
If you need to migrate an old collector version to a more recent one, please check the migration process in this article.
Devo Collector features
Feature | Details |
---|---|
Allow parallel downloading ( |
|
Running Environments |
|
Populated Devo events | table |
Flattening preprocessing | no |
Data source description
You can use the Okta collector to send this information to your Devo domain. Once the gathered information arrives at Devo it will be categorized in different tables in your domain, as you can check in the following table.
Okta services
Listed in the table below are some service names, details, and how the Devo platform treats the data.
Data source | Description | API endpoint | Collector service name | Devo table | Available from release |
---|---|---|---|---|---|
System Logs | System Log records system events related to your organization in order to provide an audit trail that can be used to understand platform activity and to diagnose problems. Often the terms "event" and "log event" are used interchangeably. In the context of this API, an "event" is an occurrence of interest within the system and "log" or "log event" is the recorded fact. | /v1/logs | logs | auth.okta.system | v1.0.0 |
Apps | Application API provides operations to manage applications and/or assignments to users or groups for your organization. | /v1/apps | apps | auth.okta.apps | v1.0.0 |
Client Application | The Dynamic Client Registration API provides operations to register and manage client applications to be used with Okta's OAuth 2.0 and OpenID Connect endpoints. | /v1/clients | clients | auth.okta.clients | v1.0.0 |
Groups | Groups API provides operations to manage Okta groups and their user members for your organization. | /v1/groups | groups | auth.okta.groups | v1.0.0 |
IDPS | Identity Providers API provides operations to manage federations with external Identity Providers (IDP). For example, your app can support logging in with credentials from Facebook, Google, LinkedIn, Microsoft, an enterprise IdP using SAML 2.0, or an IdP using the OpenID Connect (OIDC) protocol. | /v1/idps | idps | auth.okta.idps | v1.0.0 |
Users | User API provides operations to manage users in your organization. | /v1/users | users | auth.okta.users | v1.0.0 |
Zones | Zones API provides operations to manage zones in your organization. Zones may be used to guide policy decisions. | /v1/zones | zones | auth.okta.zones | v1.0.0 |
The System Log API will eventually replace the Events API. It contains much more structured data.
For more references about Okta Resources API, visit the Okta API Reference.
Accepted authentication methods
Authentication method | okta_url | api_token |
---|---|---|
API-token pair | REQUIRED | REQUIRED |
Minimum configuration required for basic pulling
Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.
This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.
Setting | Details |
---|---|
okta_url | The okta url for okta API. |
api_token | The api token for okta API. |
Vendor Setup
Getting Okta credentials
Visit Developer Okta to create an api_token and get the okta_url.
Log in with your company credentials (or sign up for a free developer account).
Click Dashboard and save the okta_url that is displayed in the top right corner (it will be used later in the config file).
On the top menu, go to API → Tokens.
Click on Create Token and enter a name for your token in the window that appears, which will be used for tracking API calls. Click Create Token.
Copy your token and click OK, got it. Note that the token will be only displayed here, so don't forget to copy it. Save it as api_token (it will be used later in the config file).
Run the collector
Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).
Collector service details
Verify data collection
Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.
This service has the following components:
Component | Description |
---|---|
Setup | The setup module is in charge of authenticating the service and managing the token expiration when needed. |
Puller | The setup module is in charge of pulling the data in an organized way and delivering the events via SDK. |
Setup output
A successful run has the following output messages for the setup module:
2024-12-04T17:15:59.197 INFO OutputProcess::MainThread -> DevoSender(internal_senders,devo_sender_0) -> Starting thread
2024-12-04T17:15:59.197 INFO OutputProcess::MainThread -> DevoSenderManagerMonitor(internal_senders,devo_2) -> Starting thread (every 300 seconds)
2024-12-04T17:15:59.197 INFO OutputProcess::MainThread -> DevoSenderManager(internal_senders,manager,devo_2) -> Starting thread
2024-12-04T17:15:59.198 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-04T17:15:59.198 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Recovering any available content from the persistence system
2024-12-04T17:15:59.201 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-04T17:15:59.201 INFO OutputProcess::OutputLookupConsumer(lookup_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.00
2024-12-04T17:15:59.223 INFO InputProcess::MainThread -> [GC] global: 60.6% -> 60.7%, process: RSS(41.81MiB -> 41.81MiB), VMS(424.86MiB -> 424.86MiB)
2024-12-04T17:15:59.225 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-04T17:15:59.225 INFO OutputProcess::OutputStandardConsumer(standard_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03
2024-12-04T17:15:59.225 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-04T17:15:59.225 INFO OutputProcess::OutputInternalConsumer(internal_senders_consumer_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03
2024-12-04T17:15:59.226 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-04T17:15:59.226 INFO OutputProcess::DevoSenderManager(standard_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03
2024-12-04T17:15:59.227 INFO OutputProcess::MainThread -> [GC] global: 60.6% -> 60.7%, process: RSS(41.55MiB -> 41.92MiB), VMS(929.13MiB -> 929.38MiB)
2024-12-04T17:15:59.227 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-04T17:15:59.227 INFO OutputProcess::DevoSenderManager(internal_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03
2024-12-04T17:15:59.228 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-04T17:15:59.228 INFO OutputProcess::DevoSenderManager(lookup_senders,manager,devo_2) -> [EMERGENCY_PERSISTENCE_SYSTEM] Elapsed seconds: 0.03
2024-12-04T17:15:59.744 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> Created a sender: {"name": "DevoSender(internal_senders,devo_sender_0)", "url": "collector-eu.devo.io:443", "chain_path": "/home/pulkit/devo/collectors/devo-collector-okta/certs/chain.crt", "cert_path": "/home/pulkit/devo/collectors/devo-collector-okta/certs/int-if-integrations-india.crt", "key_path": "/home/pulkit/devo/collectors/devo-collector-okta/certs/int-if-integrations-india.key", "transport_layer_type": "SSL", "last_usage_timestamp": null, "socket_status": null}, hostname: "2023-APAC-0049", session_id: "124538478558176"
2024-12-04T17:15:59.744 INFO OutputProcess::DevoSender(internal_senders,devo_sender_0) -> [EMERGENCY_PERSISTENCE_SYSTEM] Nothing available in the persistence system
2024-12-04T17:16:00.360 INFO InputProcess::OktaResourcesPullerSetup(okta_collector,okta#65294,logs#predefined) -> Setup for module <OktaResourcesPuller> has been successfully executed
2024-12-04T17:16:01.224 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> OktaResourcesPuller(okta#65294,logs#predefined) Starting the execution of pre_pull()
2024-12-04T17:16:01.225 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Reading persisted data
2024-12-04T17:16:01.227 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Data retrieved from the persistence: None
2024-12-04T17:16:01.228 WARNING InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Persistence will be overridden due to the retrieved state is empty
2024-12-04T17:16:01.229 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Running the persistence upgrade steps
2024-12-04T17:16:01.230 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Running the persistence corrections steps
2024-12-04T17:16:01.230 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Running the persistence corrections steps
2024-12-04T17:16:01.231 WARNING InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Some changes have been detected and the persistence needs to be updated. Previous content: None. New content: {'@persistence_version': 1, 'next_link': '', 'initial_start_time_in_utc': '2024-12-04T11:45:01Z', 'last_timestamp': '2024-12-04T11:45:01Z', 'uuid_list': []}
2024-12-04T17:16:01.233 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Updating the persistence
2024-12-04T17:16:01.235 WARNING InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Persistence has been updated successfully
2024-12-04T17:16:01.235 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> OktaResourcesPuller(okta#65294,logs#predefined) Finalizing the execution of pre_pull()
2024-12-04T17:16:01.236 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Starting data collection every 60 seconds
2024-12-04T17:16:01.236 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Pull Started
2024-12-04T17:16:02.384 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1733312761224):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
Puller output
A successful initial run has the following output messages for the puller module:
Note that the PrePull action is executed only one time before the first run of the Pull action.
2024-12-04T17:16:01.236 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Pull Started
2024-12-04T17:16:02.384 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1733312761224):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-12-04T17:16:02.384 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> Statistics for this pull cycle (@devo_pulling_id=1733312761224):Number of requests made: 1; Number of events received: 0; Number of duplicated events filtered out: 0; Number of events generated and sent: 0; Average of events per second: 0.000.
2024-12-04T17:16:02.385 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> The data is up to date!
After a successful collector’s execution (that is, no error logs found), you will see the following log message:
2024-12-04T17:17:55.904 INFO InputProcess::OktaResourcesPuller(okta#65294,logs#predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1733312874612):Number of requests made: 1; Number of events received: 255; Number of duplicated events filtered out: 0; Number of events generated and sent: 255; Average of events per second: 198.357.
Restart the persistence
This collector uses persistent storage to download events in an orderly fashion and avoid duplicates. In case you want to re-ingest historical data or recreate the persistence, you can restart the persistence of this collector by following these steps:
Edit the configuration file.
Change the value of the start parameter to a different one.
Save the changes.
Restart the collector.
The collector will detect this change and will restart the persistence using the parameters of the configuration file or the default configuration in case it has not been provided.
Note that this action clears the persistence and cannot be recovered in any way. Resetting persistence could result in duplicate or lost events.
[ Troubleshooting ]
This collector has different security layers that detect both an invalid configuration and abnormal operation. This table will help you detect and resolve the most common errors.
[ Common Logic ]
Error Type | Error Id | Error Message | Cause | Solution |
---|---|---|---|---|
InitVariablesError | 1 | initial_start_time_in_utc is not set as per the datetime_format : {datetime_format} | The date in config is not as per required format | Ensure the date format is correct. |
InitVariablesError | 2 | initial_start_time_in_utc is in the future | The date in config is not in the past period | Ensure the date period is correct. |
SetupError | 100 | HTTP Error occurred while retrieving events from okta server | Okta API call is failing | Check the credentials and ensure that the collector has the necessary permissions to access the Okta API. |
SetupError | 101 | Connecting Error occurred while retrieving events from okta server | Okta API call is failing | Check the credentials and ensure that the collector has the necessary permissions to access the Okta API. |
PullError | 300 | HTTP Error occurred while fetching siem event from okta server | Okta API call is failing | Check the credentials and ensure that the collector has the necessary permissions to access the Okta API. |
PullError | 301 | HTTP Error occurred while fetching siem event from okta server | Okta API call is failing | Check the credentials and ensure that the collector has the necessary permissions to access the Okta API. |
PullError | 302 | Some error occurred while retrieving events from Okta server. | Okta API call is failing | Check the credentials and ensure that the collector has the necessary permissions to access the Okta API. |
Check Memory Usage
To check the memory usage of this collector, look for the following log records in the collector which are displayed every 5 minutes by default, always after running the memory-free process.
The used memory is displayed by running processes and the sum of both values will give the total used memory for the collector.
The global pressure of the available memory is displayed in the global value.
All metrics (Global, RSS, VMS) include the value before freeing and after previous -> after freeing memory
Differences between RSS and VMS memory usage:
RSS is the Resident Set Size, which is the actual physical memory the process is using
VMS is the Virtual Memory Size which is the virtual memory that process is using
Activeboards
There are a number of predefined dashboards that can be downloaded in your Devo domain.
Create a new Devo Activeboard in your domain.
In the Edit mode click on the ellipsis button and select Edit raw configuration.
Open the downloaded file, select all the text, and copy it to the clipboard.
Paste the content of the file in the raw editor. Make sure you replace completely the existing configuration.
Click on Save the changes. The dashboard shows up.
Rate limits
The number of API requests for an organization is limited for all APIs in order to protect the service for all users. The number of Okta-generated emails that can be sent also has rate limits.
Okta has two types of API rate limits:
Org-wide rate limits that vary by API endpoint. These limits are applied on a per-minute or per-second basis, and some are also applied on a per-user basis. For example, if your org sends a request to list applications more than one hundred times in a minute, the org-wide rate limit is exceeded. These limits protect against denial-of-service attacks and help ensure that adequate resources are available for all customers.
Concurrent rate limits on the number of simultaneous transactions. For example, if you sent 77 very long-lasting requests to any API endpoint simultaneously, you might exceed the concurrent rate limit.
Okta has one type of email rate limit:
Okta-Generated Email Message Rate Limits that vary by email type. Okta enforces rate limits on the number of Okta-generated email messages that are sent to customers and customer users. For example, if the number of emails sent to a given user exceeds the per-minute limit for a given email type, subsequent emails of that type are dropped for that user until that minute elapses.
Rate limits may be changed to protect customers. We provide advance warning of changes when possible.
Check the following web pages for more information on Okta rate limits:
Change log
Release | Released on | Release type | Details | Recommendations |
---|---|---|---|---|
| Dec 6, 2024 | IMPROVEMENT | Improvements
|
|
| Aug 19, 2024 | IMPROVEMENT | Improvements
|
|
| Mar 18, 2024 | IMPROVEMENT | Improvements:
|
|
| Sep 8, 2023 | NEW FEATURE | New features: Added an |
|
| Jul 3, 2023 | IMPROVEMENT | Improvements:
|
|
| Apr 25, 2023 | IMPROVEMENT | Improvements:
|
|
| Nov 3, 2022 | IMPROVEMENT | Improvements:
|
|
| Jul 29, 2022 | IMPROVEMENT | Improvements:
|
|