Today, many businesses — especially global enterprises — have multiple separate applications and systems, where data crossing organizational departments or divisions can quickly become fragmented, duplicated, and out of date. It gets even more challenging when these companies acquire other organizations and add new systems to their existing ecosystem.
You can imagine, it is a challenge to understand which data is “right” and how to use that data to build a 360-degree customer view. This is a common challenge with a known solution: master data management or MDM.
One key aspect of MDM is the technical connection between all systems. If not well planned, this can get very complex and unmanageable. MuleSoft offers an efficient way to address this complexity with API-led connectivity, which builds towards an application network.
A sample scenario:
Let’s discuss the situation on the landscape shown in figure 1 below:
Figure 1: Sample Application Architecture
There are two applications, a customer web portal (CWP) and an internal-facing service application (ISA), both operating on data stored in systems A, B, C.
Without MDM, the CWP would need to take care of keeping the entity data E01 in systems A and B in sync and use entity E02 from system A only. On the other hand, the ISA uses data E01 and E02 from all three systems and needs to ensure that it has the correct information and also needs to keep the data in sync.
This complexity in processing often leads to conflicts in data consistency, which then causes errors in the CWP and ISA applications. These errors can confuse users, lower their confidence in system integrity and jeopardize the adoption of both applications.
In an API-led architecture, CWP and ISA have defined entry points via designated Experience APIs — each of which is governed with policies to control and protect the data.
Figure 2: Sample Experience API Design
This architecture scenario decouples the applications from the data repositories. They are not required to know where the data resides, simplifying the development process for these applications.
How do the Experience APIs get the “right” data? This is where master data management comes into the picture. Let’s use the common MDM patterns to discuss this in detail:
- Registry style
- Consolidation
- Centralized
- Coexistence
#1 Registry style
Figure 3: Registry Style MDM pattern
With a registry-style approach to MDM, all the data from a system is stored and managed within that particular system using its proprietary structure.
The MDM registry solution is the “master” — it keeps a directory of the records and the location of each dataset within the various systems using system-specific record IDs. The registry also defines priorities for read and write operations. In a registry style MDM approach, an API-led architecture would look like the diagram below:
Figure 4: Registry Style MDM approach with APIs
In this design, the Process APIs orchestrate access for the read and write operations. As a first step, the process APIs obtain meta-data from the registry about any specific records that are connected to the original request from the Experience APIs. This information tells the process APIs, which systems of record host the data in question, and what rules to apply when fulfilling this type of request.
For the Read operations the flow is as follows: :
- The Experience APIs request data from the Process APIs
- The Process APIs connect with the registry as described
- The Process APIs request from the System APIs the data based on information from the registry
- The Process APIs then create the response to the Experience API using the rules received from the registry
For the Write operations, on the other hand, you need to use an enhanced approach to handle potential write errors in the systems. The flow is as follows:
- The Experience APIs send the data to the Process APIs
- The Process APIs connect with the registry as described above
- The Process APIs then send the data using a transaction-safe orchestration to the System APIs, according to the rules returned from the registry
- The Process APIs update the registry with a new status of the data and then respond to the Experience APIs with the execution status
# 2 Consolidation
Figure 5: Consolidation Style MDM pattern
A consolidation MDM model provides a specialized solution with dedicated, sufficient storage to host each information’s “golden records.” The golden records are high quality data which add value to all business processes using it and aid in forming a 360-degree view.
The consolidation style MDM solution receives updates from the systems for each managed entity. Then, following technical and human-supported processes, this information is consolidated into the golden record — in fact, this capability is native to a MDM solution.
All applications using data retrieve them from a single point of storage, however, the systems are not updated from the MDM system.
Here is how the data would flow using APIs:
Figure 6: Consolidation Style MDM approach with APIs
For the Read operations the flow is like this:
- The Experience APIs requests data from the Process APIs
- The Process APIs retrieve the “golden record” from the MDM storage via the MDM system API
- The Experience APIs receive the response from the Process APIs
For the Write operations:
- The Experience APIs send the data to the Process APIs
- The Process APIs send the data to the MDM system API, which updates the MDM System
- The Process APIs than update the respective source system (and only this)
If data are updated in the System A, B or C directly, then they are pushed via the System and Process APIs to the MDM.
#3 Centralized
Figure 7: Centralized Style MDM pattern
Centralized means, all data is synced across the systems, while maintaining a centralized solution storing the “golden record.” As with the previous model, the MDM solution is responsible for the quality and storage of the golden record. What makes this pattern unique is that all systems get synced up with the most current version. One key aspect here is that the data models in System A, B and C need to be modified to be able to store the full data width.
The APIs look similar to Figure 6, and the Read process is identical to the one discussed for the consolidation model:
Figure 8: Centralized Style MDM approach with APIs
The Write process does update the golden record in the central MDM system, also identical to the consolidation model — it updates the record in the central MDM first.
And then, a data consolidation process does run:
- The MDM System API is invoked from the MDM system with updated data
- Then a Process API flow for data distribution is invoked. It would send the respective updates to systems A,B, C (via their APIs) and respond back to the MDM solution with the status
Any change of data in System A, B, C does as well invoke the system API sending the update to the MDM. It will then redistribute using the consolidation process.
Whilst the APIs for the centralized style look similar to the consolidation style in the schematic level, the internal logic is more complex to accommodate the full data synchronization.
#4 Coexistence
Figure 9: Coexistence Style MDM pattern
This pattern is a combination of consolidation and centralized approaches. This means there may be systems which require the full, consistent data record, but are not necessarily enabled for real-time synchronization.
Figure 10: Coexistence Style MDM approach with APIs
The main components look very similar to figure 6 and figure 8, with two changes: System C does not allow real time data updates. So in order to keep it in sync the dedicated Batch Sync-API manages the data flow from the central MDM solution to System C.
Making MDM plug-n-play with API-led
What is the benefit of an API-led approach in the MDM context? Imagine your organization has set up its solution using this paradigm. Then API-led helps make MDM closer to plug-n-play to include a new system D to the landscape, as you can see in Figure 11.
Figure 11: Registry Style MDM approach with APIs and new System D
It introduces the new System API D and connects it with a new version of the Process API E02. And, not to forget, the necessary configuration enhancement in the MDM solution.
Learn more about how an API-led approach can solve data management challenges by watching our webinar.
