Cloud APIs and the new spaghetti factory


With the explosion of APIs (funny that we have started using the term APIs instead of web services) what have created a monster integration challenge that I talked about in a previous post about cloud silos. However, to get an idea of how this challenge will manifest itself I thought it would be interesting to take a look at some statistics.

API Growth

Programmable web is a great resource for tracking APIs and mashups and has become the defacto registry for web APIs. The site provides a wealth of statistics tracking back to 2005 when the site was started by John Musser.

API Timeline

You can see that just 5 years ago there was only 100 odd APIs out there, before that eBay, Amazon and SalesForce were the trailblazers. Whats interesting is the acceleration and the types of APIs being released.  With the exception of Amazon APIs started out exposing application data, then social media, then digital content, then well everything else, It’s pretty amazing that by the end of this year there will probably be over 5,000 APIs  giving you access to everything from making conference calls, to storing files in the cloud, to calculating your carbon footprint with AMEE.

API as a Product

The interesting thing is that for many new companies their API is the product, Amazon Web Services is the grand daddy here, but Twilio is one of the coolest making telephony services available to everyone. is another which offers facial recognition APIs, and pretty much all of the infrastructure services like Loggly, MongoHQand SendGrid are all selling developer APIs to do stuff you used need infrastructure for.

The API is everything for the

This is a big shift in the market and a massive opportunity.  Once everything becomes accessible and addressable using APIs we can build more amazing applications in shorter time and combine things in ways that haven’t been thought of before. The cloud becomes a mega-platform for composing applications from existing application services. We’re already seeing the value in APIs, evident by looking at the number of hits these APIs receive. The API billionaires club shows how much we are starting to depend on getting our data through APIs.

API Billionaires CLub

Consumer Explosion

The explosion in smart devices such as phones and tablets are driving this usage up and will continue to skyrocket as more smart devices hit the market. Think about cars, smart meters, point of sales systems, appliances (Sun’s original vision for Java was not so crazy after all), toys, clothing, and everything in between will get connected. Some analysts estimate the number of connected devices to reach between 20-50 billion by 2020; that’s a staggering number. Does it seem unrealistic? I don’t think so, lets take a look at Netflix who’s API load has gone through the roof in recent months.

Netflix API Growth

This is a massive jump in requests driven by the fact that Netflix is now getting embedded in TVs and Blu Ray Players. This is Metcalfe’s law in action; as we add more devices to the network, the more value is achieved, both for consumers, who can get thousands of movies on demand and for Netflix that has tapped into a major revenue channel.

Spaghetti Factory

What this all boils down to is that each new API creates a new silo of data and/or functionality.  These APIs need to be integrated into applications to be useful for people. As we have seen in the past the urge is to integrate each API with the application directly.  This way of doing integration is called point-to-point integration. Anyone that has worked on projects that do integration this way often are too afraid to touch the code or decide a rewrite is needed.  The term often used for this is spaghetti code because each API is integrated directly from you application, mixing concerns of your application with no service layer to shields the application from changes in the API, or to help with authentication, error handling, monitoring and fault tolerance.

API Growth - Morgan Stanley Research

Research from Morgan Stanley in 2010 about Computer Growth drivers identify more conservative growth in connected devices but correctly assert that this exponential growth in devices presents a massive integration challenge.  The dynamics of this integration challenge is different from before. The API explosion means there are many more applications to integrate. The APIs themselves differ wildly since most favour REST (~75% according to PW) where this is no contract or schema and there is no conventional wisdom out there for building RESTful APIs. Also, many of the lessons learned in the enterprise will have to be learned again by whole different audience as more app developers build applications through composition rather than traditional coding. This new developer audience will need the right tools to avoid the pitfalls and get the job done without all the hard graft of integration.

Our goal is to build the tools for this new breed of integration development, Mule Flow and Mule Cloud Connect were designed to make cloud APIs easier to consume and compose. Mule iON is the first integration Platform as a Service (iPaaS) dedicated to providing integration between APIs and on premise applications, currently in private beta. I’ll be posting more over the coming months how we are addressing these new integration challenges.

Follow: @rossmason, @mulesoft


We'd love to hear your opinion on this post

10 Responses to “Cloud APIs and the new spaghetti factory”

  1. The API explosion is an interesting one, especially in the context of mobile apps. One big challenge though is the scalability of these APIs. While HTML5 is breeding a class of WPO technologies that help you compress and inline CSS/images and talk about CDN and Expires headers, scaling out API requests is not easily addressed. We released a few weeks ago specifically to focus on the scalability of these RESTful services. I think API, while technically is a web service, it doesn’t carry the same baggage and complexity of SOAP.

  2. Hey, we took a look a recently in the context of Mule iON. Its a very interesting approach to the problem, I liked it a lot. Congrats on the release. By the way what does WPO stand for?

  3. @ross WPO stands for Website Performance Optimization. However it’s a bit of a misnomer since the technologies only focus on the client half of the problem.

  4. Here’s a blog about this very topic: (WPO – It’s just half the battle)

  5. I recently wrote a post talking about how people can build APIs that can scale and evolve as easily as web sites

  6. Really enjoyed this post, and I’d love to learn more about what MuleSoft is up to and how it will affect and/or benefit companies like SendGrid. It’s especially interesting to me personally, since I am heading up SendGrid’s new efforts to really build and support a developer community.

    Thanks for sharing!


  7. Totally agree.
    Our website, a small one, has integration with google analytics, youtube, twitter, adsense, …
    There are thousand of different APIs and just a simple site is a mess.
    Vietnamitas en madrid

  8. […] have talked previously about the Consumer explosion, this right now is primarily driven by smart phones. Each device […]

  9. […] SalesForce, Twilio… and many more that are coming. I recommend you take a peak at this article from Ross Mason for some more context of the impact of these APIs. […]

  10. […] is iON the right solution? There are some fundamental shifts happening in I.T.  (that I’ve blogged about before) that are changing the needs of what and how we […]