SailPoint collector

[ 1 Minimum configuration required for basic pulling ] [ 2 Overview ] [ 3 Devo collector features ] [ 4 Data sources ] [ 5 Flattening preprocessing ] [ 6 Vendor setup ] [ 7 Accepted authentication methods ] [ 8 Run the collector ] [ 9 Collector services detail ] [ 10 Collector operations ]

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector. Check setting sections for details.

Setting	Details

Setting	Details
`base_url`	Base URL for SailPoint instance.
`client_id`	The `client_id` obtained from SailPoint for authentication.
`client_secret`	The `client_secret` obtained from SailPoint for authentication

See the Accepted authentication methods section to verify what settings are required based on the desired authentication method.

Overview

Sailpoint IdentityNow is an identity management solution that helps organizations manage employees' permission, digital identities, information security, data access compliance, and more on a unified portal.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`Allowed`
Running environments	`Collector server` `On-premise`
Populated Devo events	`Table`
Flattening preprocessing	`Yes`

Data sources

Data Source	Description	API Endpoint	Collector service name	Devo Table	Available from release

Data Source	Description	API Endpoint	Collector service name	Devo Table	Available from release
Account activities	Account activities entries	`/v3/account-activities`	`account_activities`	`iam.sailpoint.identitynow.account_activity`	`v1.0.0`
Events	Event related to management console	`/v3/search`	`events`	`iam.sailpoint.identitynow.event`	`v1.0.0`

For more information on how the events are parsed, visit our page

Flattening preprocessing

Data source	Collector service	Optional	Flattening details

Data source	Collector service	Optional	Flattening details
Account activities	account_activities	`No`	Account activities are flattened on the `items` list.

Vendor setup

There are some requirements to run this collector. In order to retrieve the data, you must configure a Client_ID and Client Secret for API access to the SailPoint IdentityNow Portal. Follow these steps to obtain an access token:

Accepted authentication methods

Authentication method	Client ID	Secret
`OAuth`	Required	Required

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Collector services detail

This section is intended to explain how to proceed with specific actions for services.

Account activities

Account activities are continuously fetched since the last known record time. Events are sorted by the created field. The last known event time for a given batch of record(s) and the ID associated with the latest record(s) are persisted and referenced in the subsequent pull. Account activities are deduplicated by the id field.

Account activities are flattened on the items field. Each resultant record includes all the fields of the parent object, the target child object, and fields identifying the index of the target child object.

All events are sent by default to iam.sailpoint.identitynow.account_activity.

Once the collector has been launched, it is important to check if the ingestion is performed in a proper way. To do so, go to the collector’s logs console.

This service has the following components:

Component	Description

Component	Description
Setup	The setup module is in charge of authenticating the service and managing the token expiration when needed.
Puller	The setup module is in charge of pulling the data in a organized way and delivering the events via SDK.

Setup output

A successful run has the following output messages for the setup module:

2023-02-08T08:41:12.056    INFO InputProcess::MainThread -> SailpointPullerSetup(unknown,sailpoint#111,account_activities#predefined) -> Starting thread
2023-02-08T08:41:12.057    INFO InputProcess::SailpointPullerSetup(unknown,sailpoint#111,account_activities#predefined) -> No API client found
2023-02-08T08:41:12.057    INFO InputProcess::MainThread -> SailpointPuller(sailpoint,111,account_activities,predefined) - Starting thread
2023-02-08T08:41:12.058 WARNING InputProcess::SailpointPullerSetup(unknown,sailpoint#111,account_activities#predefined) -> The token/header/authentication has not been created yet
2023-02-08T08:41:12.058 WARNING InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Waiting until setup will be executed
2023-02-08T08:41:12.059    INFO InputProcess::SailpointPullerSetup(unknown,sailpoint#111,account_activities#predefined) -> Creating API client.
2023-02-08T08:41:12.721    INFO InputProcess::SailpointPullerSetup(unknown,sailpoint#111,account_activities#predefined) -> Client fetch method get_account_activities is valid for the given credentials.
2023-02-08T08:41:12.724    INFO OutputProcess::ConsoleSender(standard_senders,console_sender_0) -> {"timestamp": "2023-02-08 13:41:12.721", "tag": "devo.collectors.out.local.info", "content": "{\"msg\": \"Client fetch method get_account_activities is valid for the given credentials.\", \"time\": \"2023-02-08T13:41:12.721867Z\", \"level\": \"info\", \"collector_name\": \"unknown\", \"collector_version\": \"unknown\", \"collector_image\": null, \"input_name\": \"sailpoint\", \"service_name\": \"account_activities\", \"module_name\": \"SailpointPuller\"}"}
2023-02-08T08:41:13.307    INFO InputProcess::SailpointPullerSetup(unknown,sailpoint#111,account_activities#predefined) -> Setup for module <SailpointPuller> has been successfully executed

Puller output

A successful initial run has the following output messages for the puller module:

Note that the PrePull action is executed only one time before the first run of the Pull action.

2023-02-08T08:50:47.043    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> SailpointPuller(sailpoint,111,account_activities,predefined) Starting the execution of pre_pull()
2023-02-08T08:50:47.043    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Reading persisted data
2023-02-08T08:50:47.044    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Data retrieved from the persistence: {'initial_start_time_in_utc': '2022-01-01T19:26:03.351000Z', 'last_event_time_in_utc': '2023-02-08T02:06:27.485000Z', 'last_ids': ['4014c5538cf74f7388cffb36dd2b8da9'], '@persistence_version': 1}
2023-02-08T08:50:47.046    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Running the persistence upgrade steps
2023-02-08T08:50:47.046    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Running the persistence corrections steps
2023-02-08T08:50:47.046    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Running the persistence corrections steps
2023-02-08T08:50:47.048    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Detected initial start time in UTC change: {'values_changed': {'root': {'new_value': '2023-01-01T19:26:03.351000Z', 'old_value': '2022-01-01T19:26:03.351000Z'}}}. Resetting state to {'initial_start_time_in_utc': '2023-01-01T19:26:03.351000Z', 'last_event_time_in_utc': '2023-01-01T19:26:03.351000Z', 'last_ids': [], '@persistence_version': 1}
2023-02-08T08:50:47.049 WARNING InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Some changes have been detected and the persistence needs to be updated. Previous content: {'initial_start_time_in_utc': '2022-01-01T19:26:03.351000Z', 'last_event_time_in_utc': '2023-02-08T02:06:27.485000Z', 'last_ids': ['4014c5538cf74f7388cffb36dd2b8da9'], '@persistence_version': 1}. New content: {'initial_start_time_in_utc': '2023-01-01T19:26:03.351000Z', 'last_event_time_in_utc': '2023-01-01T19:26:03.351000Z', 'last_ids': [], '@persistence_version': 1}
2023-02-08T08:50:47.054    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Updating the persistence
2023-02-08T08:50:47.054 WARNING InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Persistence has been updated successfully
2023-02-08T08:50:47.055    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> SailpointPuller(sailpoint,111,account_activities,predefined) Finalizing the execution of pre_pull()
2023-02-08T08:50:47.055    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Starting data collection every 60 seconds
2023-02-08T08:50:47.055    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Pull Started
2023-02-08T08:50:47.056    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Retrieving/sending account_activities event(s) having created between 2023-01-01T19:26:03.351000+00:00 and 2023-02-08T13:50:47.042987+00:00
2023-02-08T08:50:49.309    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Retrieved 250 account_activities event(s)
2023-02-08T08:50:49.462    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Flattened 250 account_activities event(s) to 2463 event(s)
2023-02-08T08:50:49.752    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Updating the persistence
2023-02-08T08:50:49.753    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1675864247042):Number of requests made: 1; Number of events received: 250; Number of duplicated events filtered out: 0; Number of events generated and sent: 2463; Average of events per second: 913.145.
2023-02-08T08:50:49.753    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Retrieving/sending account_activities event(s) having created between 2023-02-02T02:15:08.911000+00:00 and 2023-02-08T13:50:47.042987+00:00
2023-02-08T08:50:50.306    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Retrieved 49 account_activities event(s)
2023-02-08T08:50:50.332    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Flattened 49 account_activities event(s) to 486 event(s)
2023-02-08T08:50:50.391    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Updating the persistence
2023-02-08T08:50:50.392    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1675864247042):Number of requests made: 2; Number of events received: 299; Number of duplicated events filtered out: 12; Number of events generated and sent: 2937; Average of events per second: 880.217.
2023-02-08T08:50:50.393    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Retrieving/sending account_activities event(s) having created between 2023-02-08T02:06:27.485000+00:00 and 2023-02-08T13:50:47.042987+00:00
2023-02-08T08:50:50.474    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Retrieved 1 account_activities event(s)
2023-02-08T08:50:50.475    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Flattened 1 account_activities event(s) to 12 event(s)
2023-02-08T08:50:50.475    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> No events found after filtering. No further queries will be done.
2023-02-08T08:50:50.475    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> (Partial) Statistics for this pull cycle (@devo_pulling_id=1675864247042):Number of requests made: 3; Number of events received: 300; Number of duplicated events filtered out: 24; Number of events generated and sent: 2937; Average of events per second: 858.848.
2023-02-08T08:50:50.475    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1675864247042):Number of requests made: 3; Number of events received: 300; Number of duplicated events filtered out: 24; Number of events generated and sent: 2937; Average of events per second: 858.817.
2023-02-08T08:50:50.475    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> The data is up to date!

After a successful collector’s execution (that is, no error logs found), you will see the following log message:

2023-02-08T08:50:50.475    INFO InputProcess::SailpointPuller(sailpoint,111,account_activities,predefined) -> Statistics for this pull cycle (@devo_pulling_id=1675864247042):Number of requests made: 3; Number of events received: 300; Number of duplicated events filtered out: 24; Number of events generated and sent: 2937; Average of events per second: 858.817.

Events service

Events are continuously fetched since the last known record time. Events are sorted by the created field. The last known record time for a given batch of record(s) and the ID associated with the latest record(s) are persisted and referenced in the subsequent pull. Events are deduplicated by the id field.

Collector operations

This section is intended to explain how to proceed with the specific operations of this collector.