Delivering rich and consistent experiences to stakeholders and customers is one of the objectives of digital transformation. Data provides context and enables business processes to create powerful experiences. As SaaS applications proliferate within the enterprise IT landscape it inadvertently results in data silos slowing transformation initiatives. According to MuleSoft’s 2020 Connectivity benchmark report, 89% of IT leaders say data silos are an obstacle to digital transformation.
In this post, we will discuss an API-led approach to eliminate data silos and keep the application data synchronized in real-time empowering businesses to deliver consistent experiences. Specifically, we’ll walk-through an integration pattern to keep Salesforce and Procore, a cloud-based construction management software in sync, though you can repurpose this pattern with any application that supports the REST interface.
What causes data silos?
In a typical enterprise setting, one of the ways a business process gets kicked-off is by applications used by customers or customer-facing personnel. For example, a ticket can get logged in the customer support portal or a contractor can initiate a project on behalf of the customer. This data then moves downstream through one or more specialized departmental or back-office applications for further processing. By the end of the process, completion data gets duplicated multiple times creating data silos and requires deliberate effort to keep them synchronized.
While enterprises take different approaches to keep data in sync — including manual data collation, swivel chairing across screens, nightly batches, and point-to-point custom integrations — these methods are hard to scale and do not meet today’s always-on, real-time business demands.
API-led design to break data silos
To mitigate the data silos we outline a structured API-led integration pattern that is not only scalable but also reusable across the enterprise. We will take Salesforce as the upstream application used by account managers to initiate construction projects on behalf of business accounts or franchisees. Procore will be the specialized, downstream application used by project managers to manage the construction project lifecycle. The goal is the keep project data synchronized in real-time between these two applications which we will address using the following three use cases:
- Account manager creates a new project in Salesforce which is replicated in Procore.
- Project manager updates project details in Procore which is reflected in Salesforce.
- Account manager updates the project in Salesforce which is updated in Procore.
To accomplish the above use cases we recommend layered API-led connectivity as shown in the diagram below where the System API layer is designed to unlock data from systems of record — Procore and Salesforce. The Procore System API consumes Procore REST API using MuleSoft HTTP Connector Request to provide CRUD operations on the Project object. We will use the MuleSoft out-of-the-box Salesforce Connector to create and update the custom Project object within Salesforce.
The second layer is where the magic of real-time data synchronization happens. The Project Sync API listens to change events from both the applications and when an event is received the API does necessary data transformation and takes appropriate action to keep the other application in sync. Finally, the Experience layer (optional in this case) can shape the data for end-user application consumption.
Below is the screenshot of the Salesforce and Procore application side-by-side, showing Project information as applicable to the use cases listed above.
Use case implementation
Now, let’s walk-through all three use cases one by one. You can download the MuleSoft application for these use cases from Github.
1. Use case: New project creation Salesforce and replication in Procore
This use case assumes the user in the Salesforce application creates a new project and the goal now is to replicate the project in Procore. The new project create event is notified to the Project Sync API in the process layer via the Salesforce Change Data Capture (CDC). In the CDC configuration, you can click and select one or more standard or custom objects for which you want to receive notifications.
With CDC enabled for Project object, the following flow can now receive notifications using the Salesforce Connector by subscribing to the streaming channel “/data/Project__ChangeEvent.” When the Salesforce event type is “CREATE” the flow will call the Procore System API to create a new project and update the corresponding Procore Project ID back in the Salesforce record for future reference.
While most of the data field mapping between Salesforce and Procore can be easily accomplished using a simple DataWeave script there are some fields that need a lookup table. For example, the project type is entered as a string in Salesforce and transformed as a number when passing to Procore. For such cases, it is suggested to use a properties file to capture the key-value mappings and perform a lookup within the transform processor.
Pro-tip: To support data synchronization it is necessary to maintain a link using a reference ID to the individual record in the other application. In this example, a Salesforce project record will have a reference to the corresponding Procore project record and vice versa; i.e., Salesforce record ID will be stored in Procore project’s origin_id field and the Procore project ID will be stored in Salesforce project custom object’s Procore_ID__c field.
2. Use case: Project update in Procore reflected in Salesforce
Now with the project created in Procore, let’s assume the project manager has updated the project record with additional details and the goal now is to propagate those changes to Salesforce. Most modern cloud applications support Webhook that notify an endpoint based on triggers configured. For this use case, we are going to configure Procore webhooks as shown below.
With the Webhook in place, the following Procore listener endpoint will now get notified on all changes to the project object. The Procore event notification provides meta-data including the project resource ID which can be used to query the actual project object and make updates into Salesforce.
Pro-tip: When using an event-driven design to keep applications in sync you will run into a circular reference where application A updates B, which in turn results in application B triggering an update event notifying A, and so on, creating a notification loop between applications. One way to avoid this loop is to process events that are triggered by the users (through the user interface) and ignore events triggered by API updates. In this example, we use source_application_id in the Procore event meta-data object to identify the source of the event and process only events triggered by the user. The source_application_id will be API client_id if the event is due to API and null if triggered by the user via the application.
3. Use case: Project update in Salesforce reflected in Procore
This use case propagates any further changes to the project in Salesforce to Procore. It will use the same Salesforce CDC configuration and subscription as described in use case one but will look for the “UPDATE” event type and call the Procore System API with updated fields.
Pro-tip: As discussed earlier, to avoid circular reference, we only need to process events triggered by the user and ignore API triggered events. We will use the changeOrigin field under ChangeEventHeader in the Salesforce CDC event notification to identify the source of the event.
With the implementation of the three use cases, the project data will now stay synchronized in real-time between Salesforce, Procore, and the APIs can be reused and scaled to other objects between the applications. No more data silos and no need for nightly batch updates!
Summary
Data silos are a common side effect when enterprises use a plethora of SaaS applications, an obstacle for digital transformation. While manual or ad-hoc integration can help to keep data in sync it is inefficient and often doesn’t scale. This post described an API-led integration pattern to keep Salesforce and Procore project data in sync which can be repurposed to other applications across the enterprise.
For more developer resources, check out our tutorials.
