This is a guest blog from Kian Ting, an Integration Software Engineer at Christchurch City Council. The broader team consists of Kian Ting, Eden Le Comte, Deborah Murfin, Kevin Prince, and Ben Warner.
Christchurch, New Zealand, has a population of 381,800 and as a Council we want all residents to know what’s going on in and around the city.
We decided to build an event mobile app so users could:
- Save a personalised list of their favorites events.
- Get notified of any changes to their saved events (e.g. venue changes or cancellations).
- Get notified of any new events they might be interested in (based on categories they’ve subscribed to).
- Get directions to the events on their mobile devices.
That being said, making our backend systems work with a mobile app was not easy. We needed to resolve a few technical barriers (illustrated in Figure 1 below):
- How could we protect backend APIs (exposed via our Content Management System, or CMS) that didn’t have proper security or throttling capabilities?
- How could we transform different payload formats from various APIs to the format we wanted?
- How could we create a standardised way (e.g. search and pagination) for mobile apps to consume data from different backend data sources?
Figure 1
In this blog, we will describe how we solved the above challenges using Anypoint Platform by implementing a Mule application and exposing it as a secure API (Figure 2).
Figure 2
Securing and throttling backend APIs
Solving the challenges discussed above came down to addressing two issues:
- Controlling security in one place using a gateway, which allows us to filter and sort what comes in from a variety of apps
- Limiting the amount of traffic that can hit the servers at once.
We solved the problems in the following ways:
- Security on the CWP API (Common Web Platform; is the common web platform used by the New Zealand government; which is also a CMS): We used a combination of RAML (securedBy security schemes) and API Manager client-based policies to implement secureBy client_id and secret for the application.
- Limited bandwidth for CWP: We implemented a caching scope to cache requests for event data. This means that when different mobile users request the same data, the request is served with cached data (hence eliminating the need to query from backend APIs). This resulted in a reduction in the round trip request response and also took away request traffic from backend APIs.
Implementing pagination and search
Using the API means we deliver only the content that is relevant to the app user through filtering and search logic. In some scenarios, it is easier to query the data on the fly via MuleSoft (running a script in the search parameters) and search through arrays two layers deep.
Even though we do not need to alter the content of the payload retrieved from the backend APIs, we need to paginate and filter the retrieved payloads (as the retrieved payloads are always arrays of the result). Most of the technical challenges that we faced had to do with search and filtering because the data retrieved from backend APIs differ both in structure and in content. We needed a smart filtering capability.
We achieved this by implementing both the Jway JSON Path (to construct dynamic and complex JSON node navigation to the content of interest) and JEXL (to execute complex filtering expressions/scripts to filter out elements from the array payload that matches the criteria).
We then used DataWeave to call the global functions, which, in turn, references our custom Java libraries for the smart search. We also used DataWeave transformations for pagination and filtering of the retrieved payloads, as the retrieved payloads are always arrays of the result.
What’s next? Improving performance with caching
The nature of event-related data is that a lot of changes can happen, from changes to the date and time of the event to changes to the event’s location. Often, these changes happen last minute, which is why we need to build in a smart cache refresh/flush feature so that the MuleSoft caching system does not store stale data. We have figured out how to implement this feature with some help from MuleSoft support team, and this will be part of our next release.
Additional benefits in scalability
We are currently serving the proxy with 4 workers. If traffic increases, we have the flexibility to scale up to have more workers without introducing any downtime, and if traffic is low we could always scale down to save money.
In addition to the benefits mentioned above, MuleSoft also allows us to host our app’s content externally; this reduces the amount of traffic being routed through our primary website. Hosting the content via the API means it is always up to date, and means we don’t have to release app updates for content.
Learn more
If you want to see the project first-hand, then download “CHCH Events’ on your Android or Apple device to try it out. Search the events and check out all our upcoming local events. You might want to move here once you’ve seen everything we’ve got going on!
To build your own API management solution in Anypoint Platform, sign-up for a trial and follow the quick start guide.
About the team
In addition to Kian Ting, this article was written with support from the rest of the Christchurch City Council team.
Regan Fielding, Manager IT Services; Manager of the Cool Team
Kevin Prince, Lead Test Analyst; Bug Finder, Accessibility Fanatic
Deborah Murfin, Project Manager; Technical Generalist
Ben Warner, Digital Channels Advisor; The English Language Maverick
Eden Le Comte, Web Developer; JQuery Maverick
