The event-driven Web: Webhooks integration fun with Mule

Webhooks are a very simple way to tie application together on the Internet. Suppose application A wants to be informed when data changes in application B: the traditional approach consists in having application A poll B for changes. The webhooks approach turns the problem around: B gets configured so it hooks to A via HTTP calls. Whenever data changes in B (or for that matter, whenever anything of interest happens in B), B will call A to let it know. No more polling, just simple HTTP server-to-server callbacks. Webhooks is so simple that there is no spec for it: the HTTP spec is all what is needed really.

So how can Mule help? Mule can help you webhooks-enabling your applications, it can also help you handling webhooks callbacks. In this blog, we will consider a use case where Mule is used to enrich webhooks callbacks, acting as a intermediary between – reusing our previous two hypothetical servers – B and A.

The webhooks-enabled application we will consider in this example is Unbounce, a very powerful platform for creating and rolling out landing pages. If you’re a reader of the early access versions of the second edition of Mule in Action, you’ve heard about Prancing Donkey, a (alas) fictitious micro-brewery with great integration needs. Prancing Donkey has used Unbounce to create a landing page that lets people sign-up for the monthly Prancing Donkey Beer Digest:

Prancing Donkey want a direct integration of this form and its back-end system. For this, they have configured the Unbounce webhook for this particular landing page to point to their servers. Since Prancing Donkey are Mule aficionados, they will actually place Mule in the middle of the callback to perform two actions on the data coming from Unbounce:

• Transform JSON lists of values to single values: Unbounce callback JSON data uses lists for all the fields (like the email in the above landing page), but Prancing Donkey only cares about one value.
• Tentatively enrich the signup data with public profile information: for this Prancing Donkey wants to use IDlight, a free public profile lookup service.

The overall deployment diagram looks like this:

Let’s slice and dice the Studio configuration that Prancing Donkey have created to achieve this (the full XML configuration is also available). Let’s look at the first flow they got:

This flow is the one in charge of receiving the callbacks from Unbounce. It does a series of transformation before sending to a one-way VM queue and eventually acknowledging with a “200 OK” message. A VM queue has been used to decouple the reception and acknowledgment of the callbacks from their subsequent processing. This is good webhooks manners: no need to hold on Unbouce thread for the time we will enrich and persist their data. If we can perform the initial transformations visible above without issues, it means we’re happy with the data (right format) and can ack it safely. To make things even safer, the VM queue has been configured to be persistent. If Mule gets restarted amid course, processing of the messages in the “Signups” queue will restart.

Behind the scene, the configuration makes a good usage of the new and shiny Mule Expression Language (aka MEL). Here is for example single how values are extracted from the JSON objects of lists provided by Unbounce:

<expression-transformer
  expression="['email':message.payload.email[0],'ip_address':message.payload.ip_address[0]]"
  doc:name="Single Values" />

Let’s look at the second flow:

This flow is in charge of actually enriching the signup data (basically just email and IP addresses from the user who signed up) with public profile information from IDlight. It receives signups from the VM queue, invokes a private flow that deals with IDlight, transforms the resulting map to JSON and HTTP posts to the Prancing Donkey server. It also uses an until-successful routing message processor to ensure the enriched data has been delivered to Prancing Donkey (it is wrapped around the outbound HTTP endpoint).

If you’re wondering why the invocation of IDlight is performed in a private flow, here is the reason: it allows to define a local exception strategy. The idea is to let the default exception strategy kick in the above flow but define one specific to dealing with IDlight in a private flow. We’ll look at this strategy in a moment but for now let’s relish the enricher configuration that leverages MEL to safely navigate potentially missing values in the data coming from IDlight:

<enricher doc:name="Enrich with Profile data">
    <core:flow-ref name="idlightSafeInvoker"
          doc:name="Invoke IDlight" />
    <enrich source="message.payload.?name.full"
            target="message.payload.name" />
    <enrich source="message.payload.?organizations.?get(0).name"
            target="message.payload.company" />
    <enrich source="message.payload.?abouts.?get(0)"
            target="message.payload.about" />
</enricher>

Yep, that’s pretty intense but it allows us to deal with data variability, which is always a must for building robust solutions.

Finally let’s look at the last flow of the config:

After setting the Accept header to the JSON mime type, Mule performs a standard HTTP GET towards IDlight to try to retrieve a public profile. The catch exception strategy is the interesting tidbit of this configuration: if anything goes wrong with the HTTP outbound interaction (404, time-out…), the strategy will kick-in and return an empty map to the caller flow. Again, MEL to the rescue!

<catch-exception-strategy doc:name="Catch Exception Strategy">
    <expression-component doc:name="Empty Profile">
        payload = [:]
    </expression-component>
</catch-exception-strategy>

In a classic shampoo advertising manner, let’s look at the data before and after, ie. raw coming from Unbounce and after it’s been enriched:

{
    "ip_address": ["71.63.127.143"],
    "email": ["david@dossot.net"]
}

{
    "ip_address": "71.63.127.143",
    "email": "david@dossot.net",
    "name": "David Dossot",
    "company": "Dossot Software Consulting Inc.",
    "about": "Software Professional, Open Source Developer & Author."
}

Next steps

Do you want to dig more into web hooks? If yes, we recommend that you go through Jim Lindsay’s epic deck on the subject. It’s quite long: if you want to learn about what it takes to roll out your own webhooks strategy, directly hit slide 149. You’ll see then that there is a little more in webhooks than meets the eye in the first place.

Is webhooks a technology that you’re using? Or are you considering adding such hooks to your application? Would you like Mule to offer specific features to help you either consuming or offering webhooks?

Let us know your thoughts regarding this technology.