MuleSoft’s iON is a cloud-based integration Platform as a Service (iPaaS) that isn’t just for “cloud-to-premise” scenarios, but for “cloud-to-cloud” integration as well. With the current API explosion, a vast array of services is emerging which can be brought together in unimagined ways, and iON is there to enable this—like lightening between the clouds!
In this post, I’d like to demonstrate how to integrate two such cloud services. The first is Google’s well-known Gmail and the other is PivotalTracker, Pivotal Labs’ cloud solution for agile software development and project management. Developers, analysts and testers can work together by posting, editing and moving “stories” from the initial phase of gathering requirements to final delivery. Our friends at Pivotal Labs have exposed the core functionality to us as a REST API. Our solution will exploit the Cloud Connector DevKit and Mule Studio to deploy an iApp, an integration Application, to iON. Creating and commenting on stories posted to Pivotal Tracker will be as easy as sending an email!
Emails arriving in a developer’s mailbox are considered to be requests for features in the development of new software.
- If the email is new, they should be placed on Pivotal Tracker as new “feature” type stories with the subject of the email as the name of the story and the body of the email as description of the story. For future correlation, the story should be labeled with the Message-ID header of the email.
- If the email is a reply to the thread created by #1, do the following:
- If there is a story on Pivotal Tracker labeled with the first value of the References header on this email, then append the body of the email as a comment to that story.
- Otherwise, create a new story as in #1.
Overview of Solution
A Mule Flow which comprises:
- Pop3 (or IMAP) connector for consuming emails (comes with Studio)
- XSLT Transformer to create XML requests for PivotalTracker (comes with Studio)
- PivotalTracker cloud connector (we’ll create that!)
Pivotal Tracker Cloud Connector
According to DevKit instructions
For the initial question, just accept the default value.
Change the directory to pivotal-tracker-connector and run mvn eclipse:eclipse to convert it to an Eclipse project. Once in Eclipse, import it as an existing project into your workspace and then proceed to convert it to a Maven project (right-click Config -> Convert to Maven Project).
Take a look at the class generated at com.mulesoft.se.PivotalTrackerConnector
We need one such annotated method for every property which we need to configure from a flow using the config element.
We won’t exploit this, so just return true in the isConnected method:
We need one for every operation we wish to expose in the flow. @param: It is important to list all parameters passed to processor methods in the javadocs, for proper generation of docs. Otherwise the Maven build will fail.
Now, let’s spice up our connector so that it caters to the use case highlighted above! We need 4 methods:
- create-story, which should expect the story payload.
- label-story, which we need to correlate stories with subsequent emails in the thread.
- find-story-by-filter for searching
- comment-story, to add subsequent emails in the thread as comments to the original story.
Before we start coding, let’s add the following dependencies to our pom, as we will be using jersey as our REST-ful webservice client and logback for logging:
Once that’s done, we need to update doc/PivotalTracker-connector.xml.sample to facilitate doc generation:
Building the Connector for Studio
Now, while the dev-team is busy enhancing Studio to cater to our use case, we are going to do a little under-the-bonnet work and deploy our connector to Studio so that it is recognizable in our flow.
Execute: mvn install -Ddevkit.studio.package.skip=false
This will generate /pivotal-tracker-connector/target/pivotal-tracker-connector-1.0-studio.zip
This zip file contains both the binary jar for use as a dependency and icon information so that Studio can include the connector in its pallete. Unzip this file to $studioHome/dropins and restart Studio. You should see the icon for PivotalConnector in the pallete under cloud connectors.
Creating the iApp
So what is an iApp? Its an integration App that creates a composition of other applications and services to create additional value. In this case, it is a flow that integrates two or more public APIs and is deployed to Mule iON. Our app will cater fully to the use case described above by consuming emails coming into our Gmail account, transforming them into XML payloads for Pivotal Tracker and then routing them to Pivotal Tracker through our cloud connector.
Before the flow element, we need to configure our cloud connector:
And here’s the flow
Email arrives on the Inbound Pop3 Endpoint. We immediately save its From header to Session scope for use in the other flows. The Choice Routing Message Processor determines if Message-ID is present in the header to determine whether the email is new or a reply to the thread.
If the email is new, processing is delegated to the newEmail Sub-flow. Immediately we wrap the body of the email with XML markup in order to make it suitable for the XSLT Transformer. The latter prepares the XML request for Pivotal Tracker before dispatching the message to the in-memory queue, VM Endpoint.
A separate flow, createStory, consumes the transformed message on vm://createStory and immediately posts the message to Pivotal Tracker using our new cloud connector. We save the Story Id returned by Pivotal Tracker for use in other flows.
Immediately we prepare a new request to Pivotal Tracker which will label our new story with the Message-ID header of the email. We will use this later on as a corelation mechanism.
Again in a separate flow (we do this to facilitate finer-grained exception handling, which I talk about below), we consume the previously dispatched message at vm://labelStory, invoke the Pivotal Tracker cloud connector and log the successful invocation.
When our main flow determines that the incoming email is a reply on the thread, then it invokes the replyEmail sub-flow, which saves the first value of the References header to Session and subsequently dispatches the message to vm://commentStory.
If a matching story is found on Pivotal Tracker we proceed through our Choice Routing Message Processor to add the body of the email as a comment to the story again using the cloud connector after having suitably transformed the message using the XSLT Transformer.
We chose to break up our solution into separate flows in order to catch potential exceptions in the invocation of our Pivotal Tracker cloud connector. (We can’t assume the availability of the Pivotal Tracker API). Our strategy is simple. Keep retrying every five minutes up to a maximum of twelve retries. A subsequent failure is caught with our default exeption strategy. We then create a delay of 1 hour using the Quartz Endpoint before dispatching the message as it was before the exception to our Active MQ error queue. As we have a composite Message Source for each of these flows, we can consume these failed messages using the same flow which participates in the happy path.
Deployment to the Cloud!
Enter your username, password, domain name and it’s deployed!