Meet Until Successful, Store and Forward for Mule

motif

In computing like in life, not every attempt is successful the first time. A message delivery to a remote application may be impossible for a while. A particular business action may be impossible due to the temporary unavailability of an enterprise resource. The good news is that these adverse conditions may not last: all what is needed is to retry the failed operation until the issue gets resolved.

This approach is well-known in the industry. Just take a look at how emails operate: delivery between SMTP servers is attempted repetitively until it succeeds. Failure is assumed and dealt with. Following the same principles, we’re happy to introduce the Until Successful routing .

Falling back to a different resource is also a valid strategy in case of failure. This is already possible in Mule, thanks to another member of the “Successful” family, namely the First Successful routing message processor.

Use Cases

The Until Successful message processor acts very much like a store-and-forward station: it stores any MuleEvent it receives and tries to process it a configured number of times at the defined retry frequency.

As such the Until Successful message processor is able to retry:

  • Dispatching to outbound endpoints, for example when you’re reaching out to a remote web service that may have availability issues,
  • Execution of a component method, for example to retry an action on a Spring Bean that may depend on unreliable resources,
  • A sub-flow execution, to keep re-executing several actions until they all succeed,
  • Any other message processor execution, to allow more complex scenarios.

Define Success

By default the Until Successful message processor consider only thrown exceptions as indications of failure.

This said, it can also be configured to look at the result coming back from processing a MuleEvent to determine success or failure or more advanced criteria. For example, in the web world, you would use this feature to make the Until Successful message processor consider any non successful HTTP response code as a failure.

Pay attention to what you configure this message processor to execute: if it encapsulates an asynchronous sub-flow or dispatches to an in-only endpoint, it may be impossible to assert success in a systematic manner.

Examples

It’s time to take a look at the Until Successful message processor in action! This first example demonstrates how to retry sending to an HTTP-based service until success:

Here are the notable points in the above configuration fragment:

  • The Until Successful message processor relies on Mule ObjectStore for persisting the events it processes. In this example, we use an in-memory implementation: a persistent implementation would be required in order to ensure that nothing gets lost in case of a restart or crash.
  • It is configured to retry every 10 minutes for an hour. Afterwards, the message will be discarded.
  • It interacts synchronously (request-response) with the outbound HTTP endpoint in order to ensure the remote web service correctly accepted the POSTed message (ie. replied a 202 status code).

The following example shows that other flows can be retried the same way:

Notice how the Until Successful message processor has been configured to synchronously acknowledge it has accepted the inbound event for processing by returning the current message correlation ID. Sending to the “signup” VM endpoint will therefore return the correlation ID of the message whose processing by the sub-flow named “signup-flow” will be tried (and retried).

When all else has failed

If message processing keeps failing and the maximum number of retried is exceeded, the default behavior of the Until Successful message processor consists in logging the message details and dropping it.

Should you want to perform a specific action on the discarded message (for example, storing it in a file or database), it is possible to configure a “DLQ endpoint” where dropped messages will be sent to.

The above example shows how a VM endpoint can be used to receive messages that have be discarded.

SLAs FTW!

As a closing note, you’ll realize that using the Until Successful message processor will encourage you to check the availability profile defined in the SLAs of the accessed resources. Indeed, using the values found in the SLAs of the services you depend on will provide you the ideal values for the number of retries or the time between them.

This message processor is available in Mule 3.2.0 Milestone which is available from the Community download page.

 


We'd love to hear your opinion on this post


9 Responses to “Meet Until Successful, Store and Forward for Mule”

  1. How does this behave in a cluster scenario? What if the master node, which persisted the message failed and the slave node took over? Do we need to have a shared filesystem for master and slave?

  2. For this, you’ll need an ObjectStore implementation that uses some sort of shared storage (DB, distributed cache…).

  3. […] following shows a (previously shown) example of until-successful modified to use Redis as its object […]

  4. is there a documentation which shows how to configure the dead letter queue ? I want to put all the messages which were tried for max retries to go to a JMS queue

  5. I would like to use the recipient-list inside the until-successful. The problem is that if the recipient-list fails to deliver the message to some of the endpoints, the message will send to the entire list in the next try. The duplicate messages are not desired by the destinations previously received the message. I appreciate any suggestion to implement this scenario.

  6. Good question 🙂 I don’t have a ready-made answer because it depends on the transports you’re using, the type of payload, plus blog comments are not the best place to ask questions, but what I would personally try is to clone the message, setting on each clone a target URI, and use a dynamic dispatch in until-succesful. That way I’d get one message per target and each would be retried individually.

  7. I solved it as follows:

  8. I am trying to use Until-Successful in one of my flows. I am getting error during deployment while creating bean. The message i am getting is as follows.
    Cannot create inner bean ‘(inner bean)’ of type [org.mule.config.spring.factories.FlowRefFactoryBean]
    while setting bean property ‘messageProcessors’ with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘(inner bean)’:
    FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name
    ‘GetContractAndETCDecisionV2PrivateFlow’: Cannot create inner bean ‘(inner bean)’ of type [org.mule.routing.UntilSuccessful] while setting bean property ‘messageProcessors’ with key [2];
    nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘(inner bean)’: Cannot resolve reference to bean ‘objectStore’ while setting bean
    property ‘objectStore’; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named ‘objectStore’ is defined
    Code : MULE_ERROR-71999
    —————
    you mentioned that objectStore is an in-memory store readily available with Mule.
    Did i miss something OR my interpretation is wrong here?
    Do i need to implement this objectStore explicitly?
    please clarify.