Salesforce collector

[ 1 Overview ] [ 2 Data source description ] [ 3 External links ] [ 4 Vendor setup ] [ 5 Enable API access on the account ] [ 6 Authentication ] [ 7 Devo collector features ] [ 8 Minimum configuration required for basic pulling ] [ 9 Accepted authentication methods ] [ 10 Run the collector ] [ 11 Change log ]

Overview

Salesforce is a customer relationship management (CRM) solution that brings companies and customers together. It's one integrated CRM platform that gives all the departments, including marketing, sales, commerce, and service. This collector provides the possibility to integrate Salesforce with the Devo Platform making it easy to query and analyze the Salesforce data, view it in the pre-configured Activeboards, or customize them to enable Enterprise IT and Cybersecurity teams to make impactful data-driven decisions.

The Devo Salesforce Collector enables customers to retrieve data from the “Salesforce Objects API“ (a.k.a. SObjects). The collector processes the Salesforce API responses and sends them to the Devo platform which then categorizes all data received on tables along rows and columns in your Devo domain.

This collector works only with the “Enterprise”, “Performance”, “Unlimited”, “Developer”, and “http://Database.com ” Salesforce editions. Please ensure that you have one of the listed editions.

Data source description

Listed in the table below are the SObject names, their details, and how the Devo Platform treats the data. You can see that all the SObject events (JSON documents) fall under the crm.salesforceobjects technologies, except for the “event log files” processed events, which fall under the crm.salesforce one:

Product	Data source	Devo table	Collector service	Related remote endpoint	Description

Product	Data source	Devo table	Collector service	Related remote endpoint	Description
Salesforce	Account	`crm.salesforceobjects.account`	`Account`	The main endpoint is `GET https://{instance}/services/data/v{version}/{query_method}?q={query}`, where: `{instance}` is your Salesforce domain name. `{version}` is an internal variable that controls the API's versioning. `{query_method}` can be `query` or `queryAll`. `{query}` represents an SQL-like query that is composed in the following way: `SELECT @fields FROM @table Where @key >= @date_filter ORDER BY @key ASC NULLS FIRST` `@fields` represents the list of fields to be retrieved by each Collector Service. `@table` is each Collector Service. `@key` should be a date field and is used for filtering and ordering. `@date_filter` is internally controlled by the collector. Check the “Running the data collector” sections for more information.	Represents an individual account, which is an organization or person involved with your business (such as customers, competitors, and partners).
	Case	`crm.salesforceobjects.case`	`Case`		Represents a case, which is a customer issue or problem.
	Content Version	`crm.salesforceobjects.contentversion`	`ContentVersion`		Represents a specific version of a document in Salesforce CRM Content or Salesforce Files.
	Dashboard	`crm.salesforceobjects.dashboard`	`Dashboard`		Represents a dashboard, which shows data from custom reports as visual components.
	Login History	`crm.salesforceobjects.loginhistory`	`LoginHistory`		Represents the login history for all successful and failed login attempts for organizations and enabled portals.
	Opportunity	`crm.salesforceobjects.opportunity`	`Opportunity`		Represents an opportunity, which is a sale or pending deal.
	Report	`crm.salesforceobjects.report`	`Report`		Represents a report, a set of data that meets certain criteria, displayed in an organized way.
	User	`crm.salesforceobjects.users`	`User`		Represents a user in your organization.
	Event Log Files	`crm.salesforceobjects.eventlogfile` `crm.salesforce.<event_type>` if file processing is enabled	`EventLogFiles`		Represents event log files for event monitoring. Possibility to set `interval` to `hourly` or `daily` (default) from the user configuration. Risk of reaching rate limits. Enabling file processing may cause a high number of requests against the API.
	Setup Audit Trail	`crm.salesforceobjects.setupaudittrail`	`SetupAuditTrail`		Represents changes you or other admins made in your org’s Setup area for at least the last 180 days. For more information, see SetupAuditTrail in the Object Reference for the Salesforce Platform.

External links

References	Details	Link

References	Details	Link
Salesforce `SObjects`	API guidelines and documentation for Salesforce Objects.	Salesforce Developers
Event types for `EventLogFiles`	List of all the possible event types that are ingested in the `crm.salesforce.<event_type>` tables. All `event_type`s will be in lowercase and without spaces.	Salesforce Developers

Vendor setup

The Salesforce Collector works over the Salesforce API. We need to enable API access and ensure its permissions in your Salesforce account to allow the collector to get the data correctly.

Enable API access on the account

Steps	Snapshot

Steps	Snapshot
Login to your Salesforce account.
Click on the gear icon (near the user icon) and select “Service Setup“.
In the setup interface, go to the left tree and deploy “Users“ and select “Users“.
Here, find the user you want to enable the API Access for and click on its associated user profile (its corresponding value from the Profile column).
In the profile screen, click on “Edit“.
Scroll down to “Administrative Permissions“, and search for “API Enabled“.
Check it and save the changes.

Authentication

The Salesforce Collector supports both User/Password and OAuth 2 authentication methods. Choose your preferred option.

Getting credentials (User/Password)

This method needs the username, password, and security_token.

Usually, everything comes together on the same email when the user is registered in a Salesforce account, but if the security token does not come in the email or you cannot find it, follow the next steps to obtain a new one:

Steps	Snapshot

Steps	Snapshot
Login to your Salesforce account with the user that will be used to access the API.
Click on the user icon (at the top right of the screen) and select “Settings“.
In the settings left menu, find “My personal information“ and search for “Reset my security token“.
On this screen, click on “Reset Security Token“ and you will receive a mail with the new security token.

Getting credentials (OAuth 2)

To enable OAuth 2 authentication we need to create a “Connected App“ and obtain the client_id, client_secret, and redirect_uri values to use it.

Steps	Snapshot

Steps	Snapshot
Login to your Salesforce account.
Click on the gear icon (near the user Icon) and select “Service Setup“.
Type “Apps“ in the left search bar, click on “App Manager“ and create a new Connected App.
Fill the mandatory fields, check the Enable “OAuth settings checkbox” and fill the new fields as shown in the sample.
Save the changes and write down the `client_id`(Consumer Key), `client_secret` (Consumer Secret), and `redirect_uri` (Callback URL).

Getting the access token file

To get the access token file, we need to execute this script with the following steps:

Steps

Steps
Prepare your environment:
Execute the script with the following command: `python salesforce_access_token_generator.py --client_id <salesforce_client_id> --client_secret <salesforce_client_secret> --redirect_uri <salesforce_client_uri>` This script will open our predefined browser with a Salesforce login page. Log in to Salesforce and authorize the client in your domain. Once authorized, it will redirect to another URL like in the following image:
Now, we need to copy the URL and paste it into the terminal where we have executed the script, which was waiting for this URL. Press ENTER and the script will finalize the token file creation. Once finalized the execution, we should see an `access_token.json` file in the same directory as the script.
Next, we need to convert the content of this file into base64 format. This can be done using an online tool such as Base64 Encode and Decode - Online. NOTE: this is a 3rd party tool that Devo does not manage, and is provided only as an example.
The base64 encoded string is then used as the value of the `initial_access_token_base64` key in the collector’s user configuration.

Permissions

To be able to retrieve the data, we need to have some minimal permissions in the user configured for this collector. This user must be under a “Salesforce” user license and we need to provide the “standard user” profile for almost all the services/SObjects (with some exceptions) after enabling the API access.

We are now selecting from the Standard Fields Salesforce Developers

For each object they have there own standard Fields. These will all be selected all of the time. Note: If the authentactor doesn’t have permissions to view the standard or custom columns you’ll need to add Field Level Security to Visible. Salesforce Developers

The special required permissions by SObjects are specified below:

LoginHistory and User: we can use a profile inherited from “standard user“. To add the missing permissions, we just need to enable the “Manage Users“ permission in the user’s profile.

EventLogFiles: we can use a profile inherited from “standard user“. To add the missing permissions, we just need to enable the “View Event Log Files“ permission in the user’s profile.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`Not allowed`
Running Environments	`Collector Server`, `On-Premise`

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to download data with basic configuration are defined below.

Setting	Details

Setting	Details
`token_value`	Set up here your access token created in the GitHub console.
`<salesforce_user_if_using_user_and_password>`	Username to authenticate the service, if using the User/Password authentication mechanism.
`<salesforce_password_if_using_user_and_password>`	Password to authenticate the service, if using the User/Password authentication mechanism.
`<salesforce_security_token_if_using_user_and_password>`	Security token to authenticate the service, if using the User/Password authentication mechanism.
`<client_id_if_using_oauth>`	Client ID to authenticate the service, if using the OAuth 2 authentication mechanism.
`<client_secret_if_using_oauth>`	Client Secret to authenticate the service, if using the OAuth 2 authentication mechanism.
`<base_64_encoded_content_from_acess_token_file_if_using_oauth>`	The result obtained from the script described in the Getting the access token file section, if using the OAuth 2 authentication mechanism.

Accepted authentication methods

Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials configuration block.

Authentication method	Username	Password	Security token	Client ID	Client secret	Salesforce domain	Initial access token in Base64

Authentication method	Username	Password	Security token	Client ID	Client secret	Salesforce domain	Initial access token in Base64
Username/Password	REQUIRED	REQUIRED	REQUIRED			REQUIRED
OAuth 2 with all the parameters (without `access_token`)	REQUIRED	REQUIRED	REQUIRED	REQUIRED	REQUIRED	REQUIRED
OAuth 2 with the `access_token` file manually generated				REQUIRED	REQUIRED	REQUIRED	REQUIRED

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Change log

Release	Released on	Release type	Details	Recommendations

Release	Released on	Release type	Details	Recommendations
`3.0.0`	Dec 10, 2024	IMPROVEMENTS New Feature	Improvements Upgraded DCSDK from `1.12.4` to `1.13.1` Upgraded Dockerfile base image Features Added api version to be configurable, so users can update at anytime added upper and lower query parameters for requests by seconds added event log file filtering array, to choose non,some, or all events to download and send to Devo.	`Recommended version`
`2.5.0`	Jul 29, 2024	IMPROVEMENTS	Improvements Upgraded DCSDK from `1.12.2` to `1.12.4` Upgraded APi version from `57.0` to `61.0`	`Upgrade`
`2.4.0`	Jul 29, 2024	IMPROVEMENTS	Improvements Upgraded DCSDK from `1.11.1` to `1.12.2` Upgrade Base Docker Image to `1.3.0` Bug fixing Fixed the persistence configuration for the collector	`Upgrade`
`2.3.0`	Apr 9, 2024	IMPROVEMENTS	Improvements Added extra check for not valid message timestamps Added extra check for improve the controlled stop Changed default number for connection retries (now 7) Fix for Devo connection retries	`Upgrade`
`2.0.0`	Jan 8, 2024	IMPROVEMENTS NEW FEATURE	Improvements Upgraded DCSDK from 1.10.0 to 1.10.2 Fixed error in pyproject.toml related to project scripts endpoint Updated PythonSDK to version 5.0.7 Introduced pyproject.toml Added requirements.dev.txt Features: Changed the way Event Log Files were handled, converted them to json objects before sending to Devo.	`Upgrade`
`v1.7.0`	Nov 10, 2023	BUG FIXES IMPROVEMENTS NEW FEATURE	Improvements: Upgraded DCSDK from 1.9.0 to 1.10.0 Added input metrics Modified ouutput metrics Updated DevoSDK to version 5.1.6 Standardized exception messages for traceability Added more detail in queue statistics Upgrade internal dependenciesStore lookup instances into DevoSender to avoid creation of new instances for the same lookup Ensure `service_config` is a dict into templates Bug fixing: Fixed how it handles access tokens Fixed how custom objects are used Fixed which Fields it calls Fixed how the API test handles errors Features: Changed how custom services are selected Changed the way fields are selected	`Upgrade`
`v1.6.0`	Jul 25, 2023	IMPROVEMENTSBUG FIXING	Improvements: Upgraded DCSDK from 1.7.2 to 1.9.0 Ensure special characters are properly sent to the platform Changed log level to some messages from info to debug Changed some wrong log messages Upgraded some internal dependencies Changed queue passed to setup instance constructor Ability to validate collector setup and exit without pulling any data Ability to store in the persistence the messages that couldn't be sent after the collector stopped Ability to send messages from the persistence when the collector starts and before the puller begins working Ensure special characters are properly sent to the platform Bug fixing: Fixed the handling of credentials so OAuth can work again.	`Upgrade`
`v1.5.1`	May 29, 2023	IMPROVEMENTSBUG FIXING	Improvements: Added more logging when file processing is enabled for the EventLogFiles service. ELF processed file message sending refactored to be more efficient. Bug fixing: Fix tag creation to allow the creation of `crm.salesforce.{event_type}` tags when processing is enabled for the EventLogFile service (broken since v1.4.0). Statistics are now correctly calculated.	`Upgrade`
`v1.5.0`	May 23, 2023	IMPROVEMENTSBUG FIXING	Improvements: Upgraded DC SDK from version v1.6.3 to v1.7.2. Bug fixing: For EventLogFile, the collector is able to collect data hourly, instead of daily, which is the default option. Removed some columns from services Account, Case and User. These columns does not exist now in Salesforce, so they were raising errors.	`Upgrade`
`v1.4.0`	Mar 17, 2023	IMPROVEMENTS	Improvements: Upgraded IFC SDK from version 1.4.0b to v1.6.3. Automated token refresh: In previous versions, the token had to be refreshed manually, now it is no longer necessary. Added ofuscation for credentials: The collector has been improved by adding this feature to obfuscate credentials.	`Upgrade`
`v1.3.0`	Nov 29, 2022	IMPROVEMENTSBUG FIX	Improvements: Upgraded IFC SDK from version 1.4.0 to v1.4.4b. Added some extra checks for supporting MacOS as development environment The "template" supports the controlled stop functionality Some log traces now are shown less frequently The default value for the logging frequency for "main" processes hsa been changed (to 120 seconds) Added log traces for knowing the execution environment status (debug mode) Fixes in the current puller template version Docker container now exists with the proper status code New controlled stopping condition when any input thread fatally fails Improved log trace details when runtime exception happens Refactored source code structure New templates functionality functionality Upgrade Salesforce API version from version 48 to version 56. Minor improvement in refresh_token function usage Fixed bugs: Added persistence for oauth2 token to avoid `refresh the access token manually` errors Upgraded used API version to mitigate minor server side common errors such as `connection reset by peer` Upgraded IFC SDK to apply sender thread fixes to avoid sender to terminate the thread	`Upgrade`
`v1.2.0`		NEW FEATURESIMPROVEMENTS	New features: Custom services are now working for any kind of SObject. Improvements Upgraded underlay IFC SDK `v1.3.0` to `v1.4.0`. Updated the underlying `DevoSDK` package to `v3.6.4` and dependencies, this upgrade increases the resilience of the collector when the connection with Devo or the Syslog server is lost. The collector is able to reconnect in some scenarios without running the self-kill feature. Support for stopping the collector when a `GRACEFULL_SHUTDOWN` system signal is received. Re-enabled the logging to `devo.collector.out` for Input threads. Improved self-kill functionality behavior. Added more details in log traces. Added log traces for knowing system memory usage.	`Upgrade`
`v1.1.0`	Jul 15, 2022	NEW FEATURESIMPROVEMENTS	New features: New service for `SetupAuditTrail` that represents changes made by the admins in the organization’s Setup area for at least the last 180 days. Improvements: Upgraded underlying IFC SDK to v1.3.0. When an exception is raised by the Collector, the collector retries connection after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied. When an exception is raised by the Collector pull method, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied. When an exception is raised by the Collector pre-pull method, the collector retries after 30 seconds. No maximum retries are applied.	`Upgrade`