Salesforce collector

[ 1 Configuration requirements ] [ 2 Overview ] [ 3 Devo collector features ] [ 4 Data sources ] [ 5 Minimum configuration required for basic pulling ] [ 6 External links ] [ 7 Vendor setup ] [ 8 Accepted authentication methods ] [ 9 Run the collector ] [ 10 Change log for v1.x.x ]

Configuration requirements

To run this collector, there are some configurations detailed below that you need to take into account.

Configuration	Details

Configuration	Details
Enable API	API permissions should be on your Salesforce account.
Credentials	JSON file has been completed according to the option you chose.
Token file	Get the access token file.
Permissions	Salesforce user license. Standard user profile `SObject`: LoginHistory and User EventLogFiles

More information

Refer to the Vendor setup section to know more about these configurations.

Overview

Salesforce is a Customer Relationship Management (CRM) solution that brings companies and customers together. It’s one integrated CRM platform that gives all the departments, including marketing, sales, commerce, and service. This collector provides you the possibility to integrate Salesforce with the Devo Platform making it easy to query and analyze the Salesforce data, view it in the pre-configured Activeboards, or customize them to enable Enterprise IT and Cybersecurity teams to make impactful data-driven decisions.

The Devo Salesforce Collector enables customers to retrieve data from the Salesforce Objects API (SObjects). The collector processes the Salesforce API responses and sends them to the Devo platform which then categorizes all data received on tables along rows and columns in your Devo domain.

This collector will work only with the following Salesforce editions: Enterprise, Performance, Unlimited, Developer and Database.com. Please, ensure that you have one of the listed editions.

Devo collector features

Feature	Details

Feature	Details
Allow parallel downloading (`multipod`)	`Not allowed`
Running Environments	`Collector server` `On-premise`

Data sources

Listed in the table below are the SObject names, their details, and how the Devo Platform treats the data. You can see that all the SObject events (JSON documents) fall under the crm.salesforceobjects technologies, except for the event log files processed events, which fall under the crm.salesforce one:

Data Source	Table	Collector Service	Related Remote Endpoint	Description

Data Source	Table	Collector Service	Related Remote Endpoint	Description
Account	`crm.salesforceobjects.account`	`Account`	The main endpoint is `GET https://{instance}/services/data/v{version}/{query_method}?q={query}`, where: `{instance}` is your Salesforce domain name. `{version}` is an internal variable that controls the API's versioning. `{query_method}` can be `query` or `queryAll`. `{query}` represents an SQL-like query that is composed in the following way: `SELECT @fields FROM @table Where @key >= @date_filter ORDER BY @key ASC NULLS FIRST` `@fields` represents the list of fields to be retrieved by each Collector Service. `@table` is each Collector Service. `@key` should be a date field and is used for filtering and ordering. `@date_filter` is internally controlled by the collector. Check the Running the data collector sections for more information.	Represents an individual account, which is an organization or person involved with your business (such as customers, competitors, and partners).
Case	`crm.salesforceobjects.case`	`Case`		Represents a case, which is a customer issue or problem.
Content Version	`crm.salesforceobjects.contentversion`	`ContentVersion`		Represents a specific version of a document in Salesforce CRM Content or Salesforce Files.
Dashboard	`crm.salesforceobjects.dashboard`	`Dashboard`		Represents a dashboard, which shows data from custom reports as visual components.
Login History	`crm.salesforceobjects.loginhistory`	`LoginHistory`		Represents the login history for all successful and failed login attempts for organizations and enabled portals.
Opportunity	`crm.salesforceobjects.opportunity`	`Opportunity`		Represents an opportunity, which is a sale or pending deal.
Report	`crm.salesforceobjects.report`	`Report`		Represents a report, a set of data that meets certain criteria, displayed in an organized way.
User	`crm.salesforceobjects.users`	`User`		Represents a user in your organization.
Event Log Files	`crm.salesforceobjects.eventlogfiles` `crm.salesforce.<event_type>` if file processing is enabled	`EventLogFiles`		Represents event log files for event monitoring.
Setup Audit Trail	`crm.salesforceobjects.setupaudittrail`	`SetupAuditTrail`		Represents changes you or other admins made in your org’s Setup area for at least the last 180 days. For more information, see SetupAuditTrail in the Object Reference for the Salesforce Platform.

Minimum configuration required for basic pulling

Although this collector supports advanced configuration, the fields required to retrieve data with basic configuration are defined below.

This minimum configuration refers exclusively to those specific parameters of this integration. There are more required parameters related to the generic behavior of the collector.

	Setting	Details

	Setting	Details
1	`token_value`	Set up here your access token created in the Salesforce console.
2	`<salesforce_user_if_using_user_and_password>`	Username to authenticate the service, if using the User/Password authentication mechanism.
3	`<salesforce_password_if_using_user_and_password>`	Password to authenticate the service, if using the User/Password authentication mechanism.
4	`<salesforce_security_token_if_using_user_and_password>`	Security token to authenticate the service, if using the User/Password authentication mechanism.
5	`<client_id_if_using_oauth>`	Client ID to authenticate the service, if using the OAuth 2 authentication mechanism.
6	`<client_secret_if_using_oauth>`	Client Secret to authenticate the service, if using the OAuth 2 authentication mechanism.
7	`<base_64_encoded_content_from_acess_token_file_if_using_oauth>`	The result obtained from the script described in the Getting the access token file section, if using the OAuth 2 authentication mechanism.

External links

References	Details	Link

References	Details	Link
Salesforce `SObjects`	API guidelines and documentation for Salesforce Objects.	https://developer.salesforce.com/docs/atlas.en-us.sfFieldRef.meta/sfFieldRef
Event types for `EventLogFiles`	List of all the possible event types that are ingested in the `crm.salesforce.<event_type>` tables.	https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_eventlogfile_supportedeventtypes.htm

Vendor setup

The Salesforce Collector works over the Salesforce API. We need to enable API access and ensure its permissions in your Salesforce account to allow the collector to get the data correctly.

Enable API access on the account

Scroll down to Administrative Permissions, and search for API Enabled.
Login to your Salesforce account.
In the setup interface, go to the left tree and deploy Users and select Users.
In the profile screen, click on Edit. NOTE: Take into account that you will be modifying the user’s profile settings, so any other user that has this same profile will have the same settings applied.
Here, find the user you want to enable the API Access for and click on its associated user profile (its corresponding value from the Profile column).
Click on the gear icon (near the user icon) and select Service Setup.

Accepted authentication methods

The Salesforce Collector supports both User/Password and OAuth 2 authentication methods. Choose your preferred option.

Depending on how did you obtain your credentials, you will have to either fill or delete the following properties on the JSON credentials configuration block.

Authentication Method	Username	Password	Security Token	Client ID	Client Secret	Salesforce domain	Initial Access Token in Base 64

Authentication Method	Username	Password	Security Token	Client ID	Client Secret	Salesforce domain	Initial Access Token in Base 64
Username/Password	REQUIRED	REQUIRED	REQUIRED	-	-	REQUIRED	-
OAuth 2 with all the parameters (without `access_token`)	REQUIRED	REQUIRED	REQUIRED	REQUIRED	REQUIRED	REQUIRED	-
OAuth 2 with the `access_token` file manually generated	-	-	-	REQUIRED	REQUIRED	REQUIRED	REQUIRED

Getting credentials (User/Password)

This method needs the username, password, and security_token.

Usually, everything comes together on the same email when the user is registered in a Salesforce account, but if the security token does not come in the email or you cannot find it, follow the next steps to obtain a new one:

Click on the user icon (at the top right of the screen) and select Settings.
Choose My personal information in the left-menu side and search for Reset my security token.
Login to your Salesforce account with the user that will be used to access the API.
Click on Reset Security Token and you will receive a mail with the new security token.

Getting credentials (OAuth 2)

To enable OAuth 2 authentication you need to create a “Connected App“ and obtain the client_id, client_secret, and redirect_uri values to use it.

Login to your Salesforce account.
Click on the gear icon (near the user Icon) and select Service Setup.
Type Apps in the left-search bar, click on App Manager and create a new Connected App.
Fill the mandatory fields, check the Enable OAuth settings checkbox and fill the new fields as shown in the sample.
Save the changes and write down the client_id(Consumer Key), client_secret (Consumer Secret), and redirect_uri (Callback URL).

Getting the access token file

To get the access token file, we need to execute this script with the following steps:

	Steps

	Steps
1	Prepare your environment:
2	Execute the script with the following command: `python salesforce_access_token_generator.py --client_id <salesforce_client_id> --client_secret <salesforce_client_secret> --redirect_uri <salesforce_client_uri>` This script will open our predefined browser to log in to salesforce to authorize the client in your salesforce domain and will redirect to other URL like in the following image:
3	Copy the URL and paste it into the terminal where you have executed the script, which was waiting for this URL. Press ENTER and the script will finalize the token file creation. Once finalized the execution, you should see an `access_token.json` file in the same directory as the script.
4	Convert the content of this file into `base64` format. This can be done using an online tool such as Base64. NOTE: This is a 3rd party tool that Devo does not manage, and is provided only as an example.
5	The `base64` encoded string is then used as the value of the `initial_access_token_base64` key in the collector’s user configuration.

Permissions

To be able to retrieve the data, you need to have some minimal permissions in the user configured for this collector. This user must be under a Salesforce user license and you need to provide the standard user profile for almost all the services/SObjects (with some exceptions) after enabling the API access.

The special required permissions by SObjects are specified below:

LoginHistory and User: You can use a profile inherited from the standard user. To add the missing permissions, you just need to enable the Manage Users permission in the user’s profile.

EventLogFiles: You can use a profile inherited from a standard user. To add the missing permissions, we just need to enable the View Event Log Files permission in the user’s profile.

Run the collector

Once the data source is configured, you can either send us the required information if you want us to host and manage the collector for you (Cloud collector), or deploy and host the collector in your own machine using a Docker image (On-premise collector).

Change log for v1.x.x

Release	Released on	Release type	Details	Recommendations

Release	Released on	Release type	Details	Recommendations
`v1.1.0`	Jul 15, 2022	NEW FEATURESIMPROVEMENTS	New features: New service for `SetupAuditTrail` that represents changes made by the admins in the organization’s Setup area for at least the last 180 days. Improvements: Upgraded underlying IFC SDK to v1.3.0. When an exception is raised by the Collector, the collector retries connection after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied. When an exception is raised by the Collector pull method, the collector retries after 5 seconds. For consecutive exceptions, the waiting time is multiplied by 5 until hits 1800 seconds, which is the maximum waiting time allowed. No maximum retries are applied. When an exception is raised by the Collector pre-pull method, the collector retries after 30 seconds. No maximum retries are applied.	`Upgrade`
`v1.2.0`		NEW FEATURESIMPROVEMENTS	New features: Custom services are now working for any kind of SObject. Improvements Upgraded underlay IFC SDK `v1.3.0` to `v1.4.0`. Updated the underlying `DevoSDK` package to `v3.6.4` and dependencies, this upgrade increases the resilience of the collector when the connection with Devo or the Syslog server is lost. The collector is able to reconnect in some scenarios without running the self-kill feature. Support for stopping the collector when a `GRACEFULL_SHUTDOWN` system signal is received. Re-enabled the logging to `devo.collector.out` for Input threads. Improved self-kill functionality behavior. Added more details in log traces. Added log traces for knowing system memory usage.	`Upgrade`
`v1.3.0`	Nov 29, 2022	IMPROVEMENTSBUG FIX	Improvements: Upgraded IFC SDK from version 1.4.0 to v1.4.4b. Added some extra checks for supporting MacOS as development environment The "template" supports the controlled stop functionality Some log traces now are shown less frequently The default value for the logging frequency for "main" processes hsa been changed (to 120 seconds) Added log traces for knowing the execution environment status (debug mode) Fixes in the current puller template version Docker container now exists with the proper status code New controlled stopping condition when any input thread fatally fails Improved log trace details when runtime exception happens Refactored source code structure New templates functionality functionality Upgrade Salesforce API version from version 48 to version 56. Minor improvement in refresh_token function usage Fixed bugs: Added persistence for oauth2 token to avoid `refresh the access token manually` errors Upgraded used API version to mitigate minor server side common errors such as `connection reset by peer` Upgraded IFC SDK to apply sender thread fixes to avoid sender to terminate the thread	`Recommended version`
`V1.4.0`	`-`	IMPROVEMENTSBUG FIX	Improvements: Upgraded IFC SDK from version 1.4.4b to v1.6.1 Refactored source code structure Bug Fixes: When using base64 token, the access token is not refreshed automatically. This issue is fixed in this version	`Upgrade`