Today we’re interviewing Geoff Clitheroe, GeoNet Systems Development Manager at GNS Science in New Zealand. What does this beautiful but quake-prone island country have to do with Mule? This is where Geoff’s work comes into play: he’s leading a team of developers with whom he’s been building GeoNet, an advanced geological hazards monitoring system. And Mule is at the core of it.
MM. Jeff you’ve shared some very interesting blog posts about your usage of Mule in GeoNet (1 2 3). Before going into the juicy details can you summarize what GeoNet does?
GC. Thanks a lot for the opportunity to talk about how we’ve used Mule. GeoNet is a project to build and operate a modern geological hazard monitoring system in New Zealand. One of our core tasks is locating earthquakes in New Zealand. Find out more about the GeoNet Project here.
The GeoNet Project is funded by EQC (EarthQuake Commission) and operated by GNS Science. We’ve been working to replace our existing earthquake location system (Fortran running on VMS!) that requires a human to locate the earthquakes. We chose SeisComP3 as the replacement. SeisComP3 can use the high quality data from the GeoNet seismic networks to locate earthquakes automatically as fast as the data can be received and processed. The new system, GeoNet Rapid, is now in public beta.
Mule 3 is a very important part of GeoNet Rapid. We’ve dropped the time to a first earthquake location from over fifteen minutes to under a minute. The majority of the speed-up comes from removing the need for human intervention to locate the earthquake, but using Mule and faster messaging has played an important part in the time saving. Also, we can now locate all earthquakes in near real time as they happen – that’s fifteen to twenty thousand earthquakes per year. It’s why New Zealand is often known as the shaky isles!
MM. What difficulties were you trying to solve with Mule?
GC. It’s pretty typical integration. SeisComP3 produces XML messages (in QuakeML format). We use Mule to transform the QuakeML messages to a simple internal format with just the information we need for the web
(this is the good kind of loose coupling, not the kind your mother warned you about) and load them into a database (PostgreSQL + PostGIS). In production we will run multiple instances of SeisComP3 for geographical redundancy so the QuakeML messages are all going over JMS (ActiveMQ).
The website content for GeoNet Rapid is delivered using Varnish and we also use Mule and JMS to dynamically purge the cache as new content is available. For GeoNet Rapid we’re using Mule to replace system integration that I wrote using Perl and rsync (too many years ago to think about).
MM. What is the current architecture of GeoNet? Any diagrams to share?
GC. The following diagram shows how we use Mule to send QuakeML messages from SeisComP3 to the database servers used for the web.
In this case it’s very simple to run a second database client for the messages to give a complete staging environment for application deployment as well as O/S patching. This also doubles as a hot spare for the application servers.
MM. Where did you find Mule delivered the most bang for the buck?
GC. We really like open source and use a lot of it for the GeoNet Project. We do a lot of integration work and getting everyone to use a common system for integration has been very effective. It’s also important that there is a good support model for Mule, both via the community and commercial if we need it.
For me the biggest bang for buck came from switching from Mule 2 to Mule 3. I really like the simplicity of flows. I’m also frequently the “Ops” in “DevOps” and I really like the new deployment model. It’s a huge improvement over Mule 2 and has simplified things a lot for us.
MM. Can you give us any metrics, like data volume, performances, etc.?
GC. Prior to GeoNet Rapid we only sent earthquakes that were likely to be felt to the website – around two hundred per year. For GeoNet Rapid we send all earthquakes to the web site – that’s over over seven thousand so far this year. SeisComP3 uses an iterative approach for locating earthquakes. It typically releases between ten and thirty QuakeML messages for each earthquake over a five minute period.
Each QuakeML message is between 50 – 150 KB. That’s on the order of three hundred thousand messages per year. Not a lot in terms of messaging but far more than we could push using Perl and rsync! Latency for each QuakeML message is under a few hundred milliseconds and we haven’t needed to do any special tuning to Mule or ActiveMQ to achieve this. SeisComP3 is processing over 4 GB of seismic data per day in near real time (latency of a few seconds).
When there is a widely felt earthquake the website traffic can go from our typical background of tens of requests a second to over ten thousand requests a second in just a few minutes. We’ve peaked over twelve thousand requests a second. We expect this number to go down as we make earthquake locations available quicker. This is why we dynamically purge the cache content – to get the content to the users as quickly as possible.
MM. What difficulty did you encounter during the project?
GC. Earthquakes! No, seriously, in September 2010 a magnitude 7.1 earthquake occurred near Christchurch. This event and subsequent aftershocks caused deaths and severe damage in Christchurch. They also caused huge interest in the GeoNet Project. We did a lot work scaling the web hosting to cope with the increased demand. This all delayed the GeoNet Rapid project. Ultimately for development this was a good thing. It gave us a chance to switch to Mule 3!
MM. What would you do differently if you knew then what you know now?
GC. Honestly, not a lot. I wish I’d known more about JMS when we started. Now I’m really looking forward to trying RabbitMQ one day.
MM. Have you open sourced any of this work?
GC. Yes. We’ve got a few projects on GitHub. All of the Mule projects for GeoNet Rapid are open source there along with some of our other work.
MM. Thank you Geoff. We’re happy to see Mule used so successfully in the critical path of such a life-impacting system. Happy trails to GeoNet!
GC. Thanks for the chance to explain what we do and a big thanks to the Mule developers and community.
