This is the third part of a series on “Moving from RESTful to EVENTful.”
As more organizations warm up to the importance of event-driven architecture (EDA) for their IT programs, companies are moving their technologies and practices from a purely “RESTful” style to one that incorporates both REST and what I’ve been calling and EVENTful style. This hybrid approach is one that makes sense for most organizations and offers quite a bit of flexibility and opportunity. But there are challenges to making this kind of move and one of the first things to deal with are the classic problem of “naming things.” What does EVENTful mean? And is there more than one kind of EDA implementation style?
In this ongoing series, I’ve introduced the general notion of EVENTful patterns and covered the first of the four common approaches: Event Notification or EN (TK link to the second article). In this article (part three), I’ll explore a powerful approach for implementing EDA called event carried state or ECS.
What is an event-carried state?
Event-carried state messages are similar to simple event notifications we covered previously. However, instead of just providing cursory alerting information (like “Mike just logged into server-001”), ECS messages carry all the information related to the subject event. For example, instead of just broadcasting that Mike logged in to server-001 (an EN message), the ECS version of that message would contain additional important details. For instance, it might contain complete information about Mike’s account record along with additional details such as location data from Mike’s mobile device when he logged in and security-related information such as his assigned access control profile.
This makes ECS messages very powerful. Since ECS is a kind of “heavyweight” type of message, it is typically used to store data in a database or transfer large chunks of data from one system to another. For example, the login information described above might be used to update Mike’s user record in the database or add to the log data for everyone accessing your system. In fact, you can think of ECS messages much like the JSON resource objects commonly used to write data in RESTful systems. This makes ECS messages a handy approach when you are mapping out your move from RESTful to EVENTful implementations.
Another important advantage of the ECS approach is that, by carrying details of the data that was added/changed, you can reduce traffic on the network. This is different from using the EN approach where sending a simple alert with a link means that recipients of that message (possibly many recipients) will follow the supplied link to collect related data on the event. Instead, the ECS approach brings all that data to recipients in the initial response. That means less traffic on the network.
Of course, there is a trade-off. ECS messages carry more data, and that means larger messages moving around your system. Plus, by adding more information in the message, you also run the risk of carrying around data that few recipients really want or need.
Data consistency
The ECS message pattern also has some important implications for data storage services. First, when you are using ECS messages broadcast to multiple sources — and each of those sources will be storing some or all the data in the message — you introduce the possibility of inconsistency in data storage. This happens when Storage System A (SSA) processes and stores the information for the ECS message before Storage System B (SSB). When someone reads from SSA they may not get the same results as when they read from SSB. This inconsistency may only last for a few milliseconds but, in a high-traffic system that sends out thousands of ECS messages, the likelihood of an inconsistent read increases rapidly.
The challenge of synchronizing data storage at multiple locations is common in all EVENTful systems and is called the eventual consistency problem. Most storage systems built for EVENTful use have conflict and consistency algorithms built into the platform so you rarely need to do anything special when you build your EVENTful data store. However, it is important to be aware of it and learn how to handle cases where consistency may cause a problem
Data duplication
Second, in systems that rely on a single source of truth or system of record (SOR) data storage pattern, the ECS record needs to have all the possibly relevant data in order to ensure the data storage is kept up-to-date. This might mean carrying the same data in subsequent update messages even if that data hasn’t changed. Including this “unchanged data” can be important when the data storage system needs to validate the integrity of the information before saving and processing it for future use.
Summary
Event-carried state messages represent a major commitment to moving from RESTful to EVENTful infrastructure. When your organization is ready to move from simple notifications to using EVENTful implementations to read and write vital data, the ECS approach is a valuable addition to your EDA toolkit. But be aware that, when you step up to the ECS level, you’ll need to deal with the eventual consistency problem and with the possibility of data duplication and larger messages sizes.
If you want to continue to support data-writing in your EVENTful implementations and you also want to reduce the size of message payloads, you’d be better off using another style of EDA messaging: Event Streaming (ES). We’ll cover that in the next installment of “Moving from RESTful to EVENTful.”
