One of my favorite patterns from Michael Nygard’s excellent Release It! is the Circuit Breaker. A circuit breaker is an automatic switch that stops the flow of electricity in the event of a failure. This sort of behavior is also useful when integrating with remote systems.
We might want to stop message delivery on an outbound-endpoint after a certain exception is thrown. A remote system under load or the target of a denial-of-service attach is a good example. In this scenario it would be nice to automatically stop delivery for a certain period of time to not exacerbate the situation.
Configuring the Circuit Breaker
We’ll use DevKit to implement a circuit breaker and start off by using Maven to create a Mule Module for the circuit breaker processors.
Once that’s done we can define how the circuit breaker will be configured in a flow. Take a look at the following configuration:
At the top of the file we’re specifying the name of the breaker, how many “trips” until the breaker is opened and a timeout for the trip to be automatically closed. We’ll use a contrived flow for testing purposes leveraging the test:component to throw an exception.
The filter is configured before the component. If the circuit breaker is “open” then messages won’t be sent to the test:component. The trip processor on is responsible for opening the breaker if enough FunctionalTestExceptions are thrown. This configuration will stop messages from flowing to the the test:component for 5 minutes after 5 FunctionalTestException’s have been thrown.
Implementing the Circuit Breaker
Now that we know what our configuration will look like we can focus on the implementation. The full module implementation is here, but we’ll consider it one piece at a time. Its important to note that these implementations are all that’s required to supper the configuration above. No custom namespace or Spring NamespaceHandler implementations are required. We simply annotate our methods appropriately and Mule and mvn do the rest.
The trip() and filter() Methods
The trip() and filter() methods in CircuitBreakerModule.java provide the implementations for the message processors we just saw.
Let’s take a look at trip(), the method which opens the breaker, first:
This processor is called in the exception-strategy and will increment the failure count when the specified exception is thrown. If the failureCount is equal to the tripThreshold then we also set the breakerTrippedOn to the current time. getFailureCount() is retrieving its value from the ObjectStore . We’ll look at how that’s implemented shortly. First let’s see how the filter is implemented:
The filter processor will throw a CircuitOpenException if the failureCount exceeds the tripThreshold. If the tripTimeout has been exceeded then failureCount is reset and messages are allowed through. Throwing a CircuitOpenException gives us flexibility in handling messages that have been blocked. We could, for instance, place these messages on a retry queue to attempt delivery later after the breaker has been opened.
Storing the Circuit Breaker State in an ObjectStore
This implementation uses the default, persistent Mule ObjectStore to store failureCount. The following convenience methods let the processors track this state.
objectStoreMutex is a Semaphore with a single permit. This is to avoid race conditions reading from and writing to the object-store during concurrent access to the circuit breaker. Using the object-store gives us options to control the breaker, perhaps using JMX. The fact that we’re grabbing the default persistent store also means our circuit breaker’s state will be persisted across Mule instance restarts. If you’re app is running in a Mule HA Cluster then the circuit-breaker’s state will be distributed across the clustered nodes.
This blog entry hopefully demonstrated that Mule DevKit makes it easy to implement complex, schema-aware message processors in your Mule apps. This allows you to define your own integration patterns, perhaps particular to your organization, and have them peacefully co-exist with the rest of the Mule ecosystem.
