In this post I will continue talking about the various integration patterns that we used as the basis for our Anypoint Templates. The next pattern to discuss is bi-directional sync. Since bi-directional sync can be also accomplished as two, 1:1 broadcast applications combined and pointed in opposite directions, I would recommend reading my last post on the broadcast pattern before digging into this one since I will omit a lot of the same content.
Pattern 3: Bi-Directional Sync
What is it?
Bi-directional sync is the act of unioning two datasets in two different systems to behave as one while respecting their need to exist as different datasets. The main driver for this type of integration need comes from having different tools or different systems for accomplishing different functions on the same data set. For example, you may have a system for taking and managing orders and a different system for customer support. For one reason or another, you find that these two systems are best of breed and it is important to use them rather than a suite which supports both functions and has a shared database. Using bi-directional sync to share the dataset will enable you to use both systems while maintaining a consistent real time view of the data in both systems.
Why is it valuable?
Bi-directional sync can be both an enabler and a savior depending on the circumstances that justify its need. If your situation has led you to a situation where you have two or more independent, isolated, representations of the same reality, you can use bi-directional sync to optimize your processes, have the data representations be much closer to reality in both systems, and reduce the compound cost of having to manually address the inconsistencies or lack of data or the impact to your business from letting the inconsistencies exist. On the other hand you can use bi-directional sync to take you from a suite of products that work well together but may not be the best at their own individual function, to a suite that you hand pick and integration together using an integration platform like our Anypoint Platform. The reality of the situation is that no single suite, no matter how large or perfect will enable you to do everything you want, but unfortunately without the use of a proper integration platform, with best practices, a suite may get you further than a compendium of best of breed applications. With bi-directional sync you are able to let each system do what its best at while still maintaining data integrity across both of them, and if done right can buy you the flexibility to modularly add and remove systems, sometimes even multiple systems that subspecialize inside a domain like storage, CRM, or ERP. Our initial templates are for Salesforce to Salesforce integration but can be modified to be Salesforce to X or X to Salesforce and still save you considerable time.
When is it useful?
The need, or demand, for a bi-directional sync integration application is synonymous with wanting object representations of reality to be comprehensive and consistent. For example, if you want one view of your customer, you can solve that manually by giving everyone access to all the systems that have a representation of the notion of a customer. With that solution you can achieve one view of the customer at the expense of productivity, training, security, and cost. A more elegant solution to the same problem is to list out which fields need to be visible for that customer object in which systems, and which systems are the owners of those. Most enterprise systems have a way to extend objects such that you can modify the customer object data structure to include those fields. Then, you can create integration applications either as point to point applications (using a common integration platform) if its a simple solution, or a more advanced routing system like a pub/sub or queue routing model if there are multiple systems at play. With this approach you can achieve a state where each person who deals with customers can have their own view of the customer with only the information that is relevant to them. For example a salesperson should know the status of a delivery, but they don’t need to know which warehouse the delivery is at. Similarly the delivery person needs to know the name of the customer that the delivery is for without needing to know how much the customer paid for it. Bi-directional synchronization allows both of those people to have a realtime view of the same customer within the perspective that they care about.
What are the key things to keep in mind when building applications using the bi-directional sync pattern?
Filtering and Object Mapping
Bi-directional sync is very similar to a pair of broadcast applications meaning that you have the ability to scope and map the data in both directions. In our Anypoint Templates that are built on the broadcast pattern, we give you the flexibility to set the filters based on the query (since we use a pull method) and each direction had its own data mapping expression (or data mapper if we are mapping different systems). This gives you flexibility to create applications that are contextual. For illustration purposes here is an example: if you have two Salesforce instances, one which is for the California Sales team, and one for the rest of the US. If you implemented a bidirectional sync, instead of always moving all contacts from one to the other, you could set a filter like “From CA->US move all contacts that are in US but not CA” and “From US->CA move all contacts that are CA”. With these kinds of filters you will make it so that you don’t synchronize contacts which make sense in your instance, but have no value in the other instance. Similarly, if you wanted only specific fields to flow one way or another, you can accomplish that by only mapping the fields that you want to flow over. This again gives you more flexibility in managing the information that you want to synchronize.
3+ Systems
Initially with our templates we tackled a point to point, two systems solution which is of value if there are two systems that need to union their dataset. If you have multiple systems, you will need to change to a different architecture and start introducing the notion of master. Typical arrangements include something like a hub and spoke architecture where one system will be the system of record for all of the information that needs to be synchronized. For example Workday can play that role for the employee objects, or Salesforce in a B2B sales model for CRM data. Another way to do it is to decouple the systems using queues and a pub-sub model where every system will broadcast its change to a queue where every other system will look for changes that they care about. In the case of the pub-sub model you would probably want to separate your integration applications too so that the creators of those integration applications can modify and add additional applications to the pub-sub system without having to touch the ones that are already running. Hence, based on the way we came up naming convention, those would really be independent broadcast applications that amalgamate to a multi system bi-directional sync equivalent. This is one of those topics that I cannot wait to produce templates for given the need for such integration amongst our customer base looking to do integrations the right way. As soon as we have a portfolio of simpler type interaction templates we are going to start tackling some of these harder cases.
Other Considerations
There are many other considerations with bi-directional sync which apply to the migration pattern as well. I would highly encourage you to read the previous post if you have not already here.
In summary, the bi-directional sync pattern is useful when you are building integration applications that union two relevant datasets for two applications. It can enable you to use best of breed applications or fill in the gap if you have multiple systems that serve the same function but for two different sets of users. Check out our current bi-directional sync templates and feel free to leave feedback below!
