In my last post, ESB or not to ESB revisited – Part 1, I put some definition around what an ESB really is, today I’m going to describe two integration architectures, ESB and Hub and Spoke, providing the benefits and considerations for each.
Well-defined architecture makes it fairly easy to design around and scope a project
Built for Scale; no state to manage, Adding new participants does not add load on a single point in the architecture
Easy to onboard new systems by introducing a new participant on the bus
Reliable; the bus decouples applications. A failure in one system does not impact other systems
Easier to migrate legacy systems. Backend applications are abstracted by the adapter; this means you can run new and legacy systems in tandem and route messages based on your migration needs
There is up front overhead associated with the ESB architecture, including:
Defining canonical message format. This is not an easy process since you need to define a message that suits the different needs of the applications you are integrating. Typically there is a common section for all applications and then additional payloads or attachments to allow for varied content.
Defining adapter architecture. Given that you create an adapter for each application, you’ll want to make it as standard as possible so that it can be stamped out quickly for new applications. Typically the adapter takes care of data translation, validation, message routing, etc.
There is a learning curve for developers since ESB have new concepts such as endpoints, connectors, and components that are not always familiar. Also, the ESB architecture is typically asynchronous and uses messaging for communication, many developers will not have thought about systems in this way. Though with the rise of JavaScript frameworks, asynchronous systems are becoming the norm.
Recommendation
An ESB architecture is best suited for integration projects that, i) include more than a few applications and, ii) plan to be extensible allowing for the addition of more applications in the future. An ESB architecture is typically selected for strategic projects where the upfront cost of developing the architecture is part of the planning.
Hub and Spoke Architecture
Rooted in the Enterprise Application Integration (EAI) days, the hub and spoke architecture integrates applications from a central location. Each call out to a system or application returns to a central hub for further processing.
Hub and Spoke Characteristics
All systems integrated from a single location; the Hub
Usually, work application data formats directly; no canonical message
State is maintained in a shared database
To scale the architecture, the hub is clustered
Hub and Spoke Benefits
An easy architecture to get started with since there are no architecture concepts for the developer to consider
Works well for a small number of integration points (applications)
Can be scaled by clustering the hub, vertical scaling not horizontal scaling
Hub and Spoke Considerations
Single point of failure
Not good for high number of transactions
Difficult to manage over time if more systems keep getting added
Hard to scale, need to run a bigger box to get more performance; cluster for reliability
Recommendation
Hub and Spoke architecture is a great way to get started with integration, such as a proof of concepts. It is also suitable for small integration projects or to serve as an integration layer for an application.
In my recent talk at PhillyJUG, I covered these two architectures plus two others that I will describe in my next post. There were a lot of questions at my JUG presentation, so please feel free to post questions here.
I have seen a lot of users and customers looking for a solution for this following use case – you have a JMS message in transaction inside Mule and suddenly, before committing something bad happened. You want to rollback this message and retry to deliver. If unable to commit for a specific number of times redirected to a dead letter queue (DLQ).
The new deployment model for Mule 3 will provide you with a lot more deployment options compared to Mule 2. In Mule 2 a Mule instance (installation) was basically the same as a Mule application and in Mule 3.x you are able to have multiple Mule applications on the same Mule instance. This combined with the hot deployment feature provides a lot more flexibility in terms of how you are able to structure your solutions in order to minimize downtime and maximize availability. This post highlights new capabilities in Mule 3.x that greatly increase your options for how to structure your applications.
The new deployment model is extremely useful for a number scenarios. For example if you have an application that produces purchase orders that should be transformed to different formats depending on the supplier identifier within the message payload as shown in figure 1.
Fig.1 Example Use Case
The Mule 2 way to solve this would be to create three flows within the same Mule application as shown in figure 2, one flow does the content based routing and routes the message to one of the two supplier specific flows. Consider the scenario that after deployment one of the suppliers would like to change their format in some way requiring the transformation to be updated. Since all three Mule flows are contained within the same application all of the three flows will need to be redeployed even if the change is only relevant to one of them. This might be manageable for two or three suppliers but as your solution grows and you need to connect to a large number of suppliers the redeploys might become a problem.
Fig.2 Single Application Fig.3 Multiple Applications
Splitting the three flows into three different Mule applications, as shown in figure 3, allows changes to an individual supplier flow separately without affecting the message exchange with another supplier. Even if the limited downtime during the deploy may not be considered a problem it limits the risk of the deployment. Figure 4 illustrates a possible setup for the use case described earlier where an application publishes a message to a routing component that based on the content of that message routs it to the appropriate supplier specific flow via a JMS queue.
Fig.4 Deployment Model: Router – Supplier
If one of the Supplier flows consumes a lot of resources or new SLA requirements are introduced, that specific part of the solution can be moved to a dedicated server without having to update the code in any way. This task could easily be managed by an operations team without having to involve any developer in the process.
Figure 5 illustrates how the “Supplier B Flow” have been separated out to another server.
Fig.5 Deployment Model: Router – Supplier A – Supplier B
To summarize, when designing and structuring Mule applications, it’s important to consider the frequency with which parts of the solution will need to change and grow. Failure to do so may lead to monolithic applications that are extremely hard to manage and troubleshoot. The benefit of partitioning flows and services that may change or grow gives much more deployment flexibility should your architecture need to change to result in increased load or a change in requirements.
In part 1 of this post, I gave an overview of BPM, Rules, and CEP and the way they compliment an integration platform such as Mule. Now let’s take a look at what Mule has to offer for integrating with some of these tools.
Magento eCommerce Platforms provide the scalability, flexibility and features for business growth. Magento provides feature-rich eCommerce platforms that offer merchants complete flexibility and control over the presentation, content, and functionality of their online channel.
Now you can easily integrate Magento with your other applications using Mule. The Magentoconnector exposes the common MagentoAPIs for Order tracking, Customer management, Stock Management, Invoicing, Product Management, Categorization and more. The Mule connector offers a Java interface to the API and integration with Mule Flow.
I wrote a post a while back, To ESB or not to ESB, about when to use an ESB and when not to. It got a fair bit of pick up, and I’ve had a lot of people reach out to me about it with specific use cases. It got me thinking it’s time to revisit the topic and provide more practical information about your options for integration.
This is a multi-part post where I’m going to start by defining what an ESB really is, offering up alternatives for the integration landscape and then finish off with how Mule fits into this landscape.
ESB is fundamentally an architecture; It is a set of rules and principals for integrating numerous applications together over a bus-like infrastructure. ESB products enable users to build this type of architecture but vary wildly in the way that they do it and the capabilities that they offer.
The concept of the ESB architecture is that you integrate different applications by putting a communication bus between them and then enable each application to talk to the bus. This decouples systems from each other, allowing them to communicate without having a dependency or knowledge of other systems on the bus. The concept of ESB was born out of the need to move away from point-to-point integration, which becomes brittle and hard to manage over time. Commonly, the ESB is often drawn as follows:
The ESB diagram above is technically overly simplistic. Let’s take a look a the tenets of an ESB architecture to highlight what is missing from this picture.
The ‘bus’ concept decouples applications from each other, this is usually achieved using a messaging server like JMS or AMQP.
The data that travels on the bus is a canonical format and is almost always XML
There is an ‘Adapter’ between the application and the bus that marshals data between the two
The adapter is responsible for talking to the back-end application and transforming data from the application format to the bus format. The adapter can also perform a host of other activities such as message routing transaction management, security, monitoring, error handling, etc.
ESBs are generally stateless; the state is embedded in the messages passing through the bus.
The canonical message format is the contract between systems. If you speak the canonical format, you can communicate with other applications.
The ESB diagram above has no mention of messaging servers or adapters between applications, to be more accurate we can add the next level of detail:
Where is the Bus?
I find that the ‘B’ in ESB confuses people since there is no actual bus, at least not in the way people think about a bus. In reality, there is a messaging server such as JMS, AMQP, Tibco RV or WebsphereMQ used as the communication channel. The messaging server is responsible for delivering the messages between application adapters. So in truth, the Bus is real as the set of the queue through which data can be sent and received. These act like well-defined addresses for the Adapters to communicate with each other. Typically an application adapter will have an inbound and outbound queue to receive and send messages.
To build on our previous example the ESB architecture looks something like this:
The ESB Adapter is really where a lot of the magic happens. This is where data marshaling, transformation, security, transaction management and mediation is handled. There are usually another set of capabilities that an ESB product provides for managing to monitor and governing the implementation.
ESB is just one architecture
We see a number of different architectures that people build to solve their integration needs. These range from very simple requirements to full-blown strategic architectures for integration. In part 2 of this post, I will explore other architectures used for integration such as Hub and Spoke, Grid, Pipeline, Enterprise Service Network and simply embedding integration inside a WebApp. I will provide some context of the pros and cons of each approach.
In the meantime, I hope this has helped put some clarity around what an ESB architecture is as well as the principals behind it.
The new MuleForge works with different project hosting platforms, it is now a repository of all things Mule rather than a project hosting service. I recently migrated the Maven Mule Plugin from our old MuleForge infrastructure to GitHub. This was quite a journey and I had to trawl the web quite a bit to gather all the required steps to make the change. So here’s the full story, in case anyone wants to do the same.
Migrating from SVN to git
The very first thing I did was to migrate the source. There are plenty of (differing) tutorials out there but the “Migrating to Git” chapter in the Pro Git book got me started quite nicely.
There are some more steps you have to perform, though. The instructions tell you to convert all the SVN tags into git tags. But you have to actually push them so they appear on GitHub:
The Mule team is always asking ourselves “how can we make developers more productive?” and “what would make the development process easier?” The recent beta of Mule Studio certainly was a case in point, but Mule offers a number of useful developer tools for use throughout the development, test and rollout process. Folks may be familiar with a few of these. Others, perhaps you’ve missed.
We thought we’d offer 45 minute technical webinar with offer an overview of these development tools and how they can improve the ease of developing, testing and maintaining your applications.
Topics covered In this 45-minute technical webinar include:
Best practices for management and configuration of development and test environment
An in depth demonstration of using the Mule Flow Analyzer in remote environments for problem investigation and resolution including inspection of message payloads
Speakers: David Eason, MuleSoft Pre-sales Engineer
Logistics: Date: June 9, 2011 Time: 10am PT / 1pm ET / 6pm BST / 7pm CET (find your time zone)
Once you have a working Mule ESB application you may be wondering how fast it can run. Here we will discuss a simple method for measuring the throughput of your application using Apache JMeter.
Bear in mind there are many ways to improve performance (simple changes can yield great performance boosts). We will explore them in greater detail in a follow-up blog post covering Mule application tuning.
