A primitive look at digital integration: Developing a common language for connectivity concepts

Integrating software is almost as old as writing software. From ETL to EAI to SOA, the software engineering industry has continually sought new integration methods to get quick access to data and functionality, avoid duplication of effort, and deal with federated ownership of applications. In the current cloud-native, highly-distributed software landscape, even the simplest mobile app is built on a complex, integrated back end. Web APIs have become ubiquitous in this landscape, but other methods of integration — asynchronous events, streaming, platform protocols — are gaining popularity, leading to a convergence that Mike Amundsen is detailing in his latest blog series. With such a crowded playing field of techniques and technologies, and with integration playing such a critical role in digital business, I think it’s worth taking a look at the language of integration. Having common terminology for common concepts will help the API-event convergence go more smoothly.

A primitive language

Where does one start to define a language for integration? Aren’t there already enough terms and buzzwords out there? Absolutely. The problem is that we have too many words that might mean the same thing depending on context. In this post, I will propose a set of integration primitives; that is, some basic terms that refer to concepts that span the integration landscape. Understanding these primitives should help you relate a concept that may be known by one term in, for example, in the web API world, and a different term in the world of events. I’ll examine four categories of primitives: integration system primitives, interaction type primitives, interaction pattern primitives, and interaction composition primitives. Comprehensive work has been done before to decompose the integration landscape — most notably by Gregor Hohpe and Bobby Woolf and their Enterprise Integration Patterns — so the focus here is to provide a simple set of building blocks to help architects and designers.

Integration system primitives

The problem space of digital integration is a system made up of interrelated parts. We have different names for the parts and the relationships depending on the scope of the integration. It could be web applications sending HTTP payloads over the internet. It could be microservices distributing Kafka-based event streams to one another. It could be batch jobs pushing files to trigger new batch jobs. It could be a combination of all three. Given that integration is a systems problem, we can use systems engineering terminology for the most basic primitives. In an integration system, components relate to each other through interactions facilitated. Interactions consist of one or more messages, and are communicated by and through interfaces.

Using this definition, we can see that the web applications, microservices, and batch jobs from the above examples are all components within their contexts. The HTTP payloads, Kafka events, and batch files are all message-based interactions, using protocol-specific interfaces. Applying this to the converged world of web APIs and events, a single microservice may consume APIs from another microservice, consume events from a third, and produce events that are consumed by a fourth. This microservice is not just an API consumer, an event consumer, or an event producer; it is a single component playing these multiple roles. This distinction is significant for the design of the overall integrated systems, since it does not artificially constrain design choices at the system level. Similarly, modeling interactions between these components without premature concern over protocols helps avoid unnecessary constraints.

Interaction type primitives

When modeling an integrated system, you should start by attacking the complexity of the business problem space rather than worrying about protocols and implementation details. In the converged world of web APIs and events, I think this is the most important shift integration architects and system designers need to make. API-first architects might start by modeling resources, consumers, and data sources. Event modelers might use event storming to enumerate domains and events. A unified approach is to consider the components involved in the problem space, determine what interactions need to take place between them, and then classify those interactions based on their impact on the state of the system.

In my experience, there are three interaction type primitives with respect to the state of the system: queries that do not change the system state, commands where components collaborate to change the system state, and events that communicate the changed state. I wrote about this classification in a previous blog post, and think it also belongs as part of the integration primitives.

Interaction pattern primitives

Whereas interaction type primitives describe the intent of interactions and their impact on system state, interaction pattern primitives describe the way messages will be packaged and delivered as part of an interaction. From a design perspective, thinking through interaction patterns is a good way of identifying system constraints early, especially around the degree of coupling between components.

Interaction patterns are determined by the number of messages packaged in the interaction, as well as how many components are involved. Some of the most common examples are request-reply, one-way, multicast, batch, and stream, with details on each in the table below. This list is by no means exhaustive. In practice, people often conflate interaction types, patterns, and underlying protocols. For example, people assume that all “commands” are “one-way”, or that all events must be delivered over an asynchronous protocol. Separating out these concepts helps to identify the processing and policies that should be applied to each interaction.

Interaction composition primitives

In addition to patterns of individual interactions, there are also common patterns for groupings of interactions. One of the big benefits an organization can achieve through an API-led architecture is to enable composability of business capabilities for different user experiences and business process flows. There are different ways of composing those capabilities, including three common patterns. The simplest form of composition is aggregation. This is a form of stateless composition where a component receives a single query request, then executes multiple queries to other components and assembles their responses to formulate a reply. Orchestration is the stateful coordination of interactions, where the flow may change depending on intermediate results. The third composition pattern, choreography, is the reactive coordination of component actions based on interaction triggers and resulting events. An enterprise’s distributed integration architecture — or even a single complex business process — may feature all three.

Identifying the right composition pattern helps to determine appropriate architectural choices and implementation decisions. For example, aggregation — analogous to the “backends for frontends” pattern — is common in optimizing communication for mobile and web applications. This can be done through middleware without the need for complex business logic. Orchestration, on the other hand, typically includes business rules, and therefore it may be better implemented within a component that is business-aligned, functioning more as a composite service than as a centralized workflow. Choreography likely requires business-agnostic event distribution capabilities in conjunction with business event-handling logic within each participating component. A common approach in a microservice architecture is to have orchestration within the scope of a microservice, with choreography between microservices.

Integration primitives in practice

There are many benefits to breaking down the integration landscape into these primitives. Firstly, it’s useful to develop a common language in order to overcome so many overloaded terms, especially with the web API and event worlds colliding. Secondly, we’ve already seen examples of how the use of primitives and their associated patterns can help identify appropriate processing and policies to apply. The last benefit, which may be the biggest, is that using these primitives as labels opens up a new approach to designing and visualizing integrated systems of software components and highly-distributed interactions. That approach will be the focus of a future blog. Until then, I hope these ideas have been thought-provoking, and please chime in with your own thoughts in the comment section below.

Thanks to Craig Bayley for contributing to this post.

Learn more about the world’s leading integration platform: Anypoint Platform.