API Best Practices: Hypermedia (Part 4.2)

Reading Time: 8 minutes

This is part four, sub-series 2, of the API design best practices series. Jump to sub-series 1 of the hypermedia sub-series.

The Harsh Reality of the State of Hypermedia Specs

Hypermedia sounds great in theory, but theory only goes so far. Where hypermedia really shines, or completely fails, is in implementation. Unfortunately, as hypermedia is still a relatively new aspect of web based APIs, there isn’t one specified way of doing things. In fact, you’ll find that even some of the most popular APIs operate completely differently from each other.

After all, there are several different hypermedia formats available for API providers to choose from. Just for starters there is HAL, Collection+JSON, JSON-LD, JSON API, and Siren! But the list doesn’t stop there, as some popular APIs have even elected to create their own format.

For example, while PayPal’s API closely mimics the JSON API format, it goes a step further and adds a method property (not part of the JSON API spec), creating a more flexible spec and transforming it from being resource driven to being action driven:

This has the potential to let developers create a more agile client based on the actions (and methods) available to them. However, for developers not familiar with PayPal’s format, but familiar with JSON API this may cause slight confusion (although it should be quickly remedied by reading their docs).

VerticalResponse, on the other hand, has taken a different, albeit interesting approach. For their API they likewise start with the basic JSON API format, but for some reason decided against the universally accepted “href” or Hypertext Reference property, instead opting to use “url” or the uniform resource locator as the link URI identifier:

Personally, I would recommend staying with the uniform “href” attribute as it denotes a reference to a hypertext link and is not as exclusive as an URL- which is not (although it commonly is) to be confused with URI. But you can read more on that here.

On the other hand, Amazon’s AppStream API, Clarify, Microsoft’s Lync, and FoxyCart all prefer to follow HAL or the Hypertext Application Language format. HAL provides a simple format for nest-able links, but like other specs omits the methods property as included by PayPal, making theirs truly unique in that sense:

However FoxyCart takes it one step further, not only taking advantage of hypermedia, but offering multiple formats for their clients to choose from, including HAL+JSON, HAL+XML, and Siren.

This, however, highlights once again one of the biggest challenges with hypermedia driven APIs, the abundance of ideas and specs available for execution. While on one-hand I believe that by supporting both XML and JSON, as well as having multiple JSON formats FoxyCart is by far the most flexible (format wise) of the APIs, not having a singular standard for each language does present the challenge of forcing developers (and hypermedia clients) to support multiple formats (as they integrate more and more hypermedia APIs), while also having the understanding that not one format meets every API’s needs.

The good news, is that despite these growing pains, we are starting to see companies adopting certain specs over others, while also identifying areas for improvement (such as with PayPal’s adding of methods to JSON API). Next week we’ll take a look at some of the most popular formats out there in-depth, keying in on the strengths and weaknesses of each.

But it’s important that as you build your API, you understand WHY you are building it the way you are. And this extends into how you build your hypermedia links, and whether or not you choose to take advantage of a standardized format (recommended), or venture off on your own to meet your developers’ needs. One of the best ways to do this is to explore what others have done with their APIs, and learn from their successes, and their mistakes.

It’s also important to consider where technology is going. And as more and more formats become available and change in popularity, it may be smart to follow FoxyCart’s lead – taking advantage of the spec that best meets your developers’ needs, but also keeping the link format decoupled enough from your data that you are able to return multiple formats based on the content-type received. Something that will allow you to take advantage of this best practice, while also being prepared for whatever the future may hold.

Continue Reading on Hypermedia →

  • Using Hypertext Links in your response
  • Existing Hypermedia Specs (HAL, JSON-LD, etc)

API Best Practices: Hypermedia (Part 4.1)

Reading Time: 19 minutes

This is part four, sub-series 1, of the API design best practices series.

What is Hypermedia

One of the challenges to implementing and correctly using hypermedia in your REST API is first understanding what hypermedia is, and what it means to use hypermedia as the engine of application state (HATEOAS).

The term “hypermedia” was coined back in 1965 by Ted Nelson, and over the years has dominated the technology industry. Hypermedia, in its most basic sense is an extension of hypertext – something you may recognize from HTML. Hypertext is essentially text that is written in a structured format and contains relationships to other objects via links. If you think of a webpage, using HTML – the Hypertext Markup Language – text is then interpreted by a browser to become a webpage, or an interactive environment capable of doing more than just providing a blob of text. However, to be truly interactive and guide the user, the crucial component of all of this is links, something we use everyday when surfing the web.

Enter hypermedia, again just an extension of the term hypertext, hypermedia includes images, video, audio, text, and links. In a REST API, this means that your API is able to function similarly to a webpage, providing the user with guidance on what type of content they can retrieve, or what actions they can perform, as well as the appropriate links to do so.

Essentially, the easiest way to take advantage of hypermedia in your API is to provide valuable information to direct the user or client to the next possible actions they can take based on the object (whether it be a collection, or item within the resource) or “page” they are on via links.

Another way to think of it is to bring back the Hypercard application from the early Apple days (we’re talking Macintosh Classic). With this application you could create “cards” or slides with links that performed different functions. Clicking on link might play a sound, another a video, or another would take you to a different card. On each card you were presented with certain links (whether in the form of a button or text) for the available actions to you on this card. This is exactly how hypertext links work within RESTful APIs– they are designed to take you to the other components of the API, your other “cards.”

By adding these links, because REST is stateless, we are providing the client with a method for determining the state of the current item/ collection that they are on, as well as giving them the roadmap or engine for what they can do with it. In other words, we are using Hypermedia as the Engine of the Application’s State, or implementing HATEOAS.

The HATEOAS Debate

Currently, there is a lot of disagreement around hypermedia’s place in RESTful APIs– with Dr. Roy Fielding saying it is critical to REST, and without it your API cannot be RESTful; and hypermedia skeptics arguing that you are doing nothing more than bloating your API and making it more difficult to use under the guise of usability.

This means you will find many different stances on the web today, ranging from Peter Williams explanation of how hypermedia has helped him, Jeff Krupp’s rant against hypermedia, and even Kin Lane’s summary of the whole debate.

But what it really comes down to isn’t what do others think about hypermedia, as most major frameworks for building APIs now include hypermedia specs such as HAL, JSON-LD, JSON API, Siren, or Collection+JSON, but rather what benefit does hypermedia offer- and is that benefit enough to outweigh the cost/ disadvantages of incorporating hypermedia.

The Claims For and Against Hypermedia

Some of the most common arguments for and against hypermedia include:

Hypermedia creates more work

This is absolutely true. Adding hypermedia or hypertext links to your API output does require more work – and more thought to go into your API. Going back to the Planning Your API post in this series, hypermedia explains how your API works together, what resources work with which, and what you can do with a collection, or specific items in that collection. This will add to both the workload and the time it takes to build your API. However, by having these relationships already drawn out as suggested, the additional time required will be minimal.

Hypermedia will require more Data-Transfer

Again, absolutely true. By adding the links to your response you are increasing the amount of data that needs to be sent back, and slowly down the responses ever so slightly. However, for MOST APIs the amount of data being added will be minimal and the data/ time difference result will realistically be unnoticeable. What you will do, however, is help avoid unnecessary calls that are not accessible to a particular item, but may be available to other items. See the next section for more on this.

You are creating an API for a client that doesn’t exist

Another argument against APIs is that unlike web browsers, there are no browsers for APIs. Likewise a valid argument, however, one I personally believe falls short as like the web we are seeing advances in APIs and recently an explosion of API exploration tools ranging from API Consoles to the API Notebook. As more and more APIs are developed there will be more and more emphasis on being able to explore them pragmatically.

Another reason I believe this argument falls short is that by utilizing hypermedia you allow the developers USING your API to build their systems around the information you provide them, creating a more dynamic application that relies on the available actions YOU return to them instead of creating rigidity by hardcoding what actions the user can take from their application in relation to your API and returning errors when those actions fail because they simply were not available to begin with.

For example, for one user you may be able to:

  • Edit
  • Suspend
  • Delete

But for another user who has been suspended, maybe you can only:

  • Restore
  • Delete

Because this is dynamic data the developer has no way to determine what actions are available and which ones are not, unless they dig through and try to decipher the user’s data which has been returned- making assumptions about your architectures rules which may or may not be correct, and may change with future development. Hypertext links in this case allow the developer to rely on YOUR rules and architecture, rather than trying to mimic it with their own.

Nobody knows how to use Hypermedia

There is some truth into this. Despite existing since 1965, many developers are flustered when first running into hypermedia, choosing to ignore it or misunderstanding how to use it (i.e. hard-coding the links). What can help with this, however, is more APIs taking advantage of hypermedia (as being encouraged in the most popular development frameworks), and simple explanations within documentation explaining how the hypertext links work (just as we had to explain how to do a GET, POST, PUT, PATCH, and DELETE when REST first came out). However, unlike many specs and challenges out there, utilizing hypermedia from a client has a very fast learning curve, and most developers are able to implement it quickly and without any real issue once they understand what it is designed to do. And once implemented correctly, many developers appreciate knowing that as resource URIs change or additional query parameters are required of them, that their application will be able to “automatically upgrade” to these new requirements without any work or concern on their part.

Hypermedia makes your API MORE Flexible

Absolutely true. By adding Hypermedia you are able to add new features more seamlessly- making them immediately available to your users. It also gives you the power to change certain aspects of your API (i.e. changing resources, requiring additional GET parameters) without necessarily breaking backwards compatibility IF implemented correctly by your users.

Hypermedia prevents APIs from breaking

Absolutely false. As mentioned above, Hypermedia does make your API more flexible, but does not excuse poor design or allow you to break backwards compatibility. One of the reasons for this, is even if you have a purely hypermedia driven API, such as Stormpath‘s, developers will still hardcode certain URLs to avoid having to make multiple calls. For example, they may access the /users resource directly to get a list of all users instead of starting from the gateway or entry point of your API and working up to that point through multiple calls. While this technique does save you (and them) multiple API calls, it does limit your ability to change common or most frequently used resource URIs.

Hypertext links also do nothing for backwards incompatible changes in data– although you could hypothetically use them as a form of versioning.

Hypermedia replaces Documentation

Hypermedia does increase discoverability of your API’s resources and methods, however, it does not replace documentation as it doesn’t provide method information, schemas, or examples. Documentation also serves as a preview to your API, letting developers understand how it works, and potentially make sample calls using an API Console before implementing it into their application. Hypermedia also does not play a solid role in debugging the implementation of the API when things go wrong. For this reason, having well written, informative documentation is vital to any API. In fact, many RESTful hypermedia specs including HAL grant you the ability to embed documentation links for quick reference by developers.

Hypermedia is a Best Practice

Where hypermedia shines is in its ability to create a flexible API that provides dynamic data for developers based on your architecture and not their own. In essence, it provides a shortcut for developers that allows them to utilize your API to its fullest without having to rely solely on documentation and writing rules that may or may not not be consistent with those in your application. Like when building a simple website, one may not see the advantage of Hypermedia right away, but as your API grows and becomes more complex you will find that it becomes a powerful convenience layer, one that will help developers better understand and navigate your API, and one that may prevent you from having to version your API for certain changes that would otherwise be backwards incompatible — if implemented correctly by your users. As Peter Williams explains in his post, hypermedia turned out to be a saving grace. And as a key component of REST as defined by Dr. Roy Fielding, it remains a best practice to implement.

In Summary

Hypermedia is often misunderstood in regards to APIs, but essentially it functions exactly like links on a webpage. And while the technology is both praised and criticized, it does provide an array of short and long-term gains. These gains, I believe, become more and more noticeable the larger and more complex your API becomes. However, the best features of utilizing HATEOAS, in my opinion, are yet to be seen as technology and API exploration tools continue to advance.

Be sure to join us next week as we take a look at implementing hypermedia into your API and the current specs out there to help you do so.

Continue Reading on Hypermedia →

  • Using Hypertext Links in your response
  • Existing Hypermedia Specs (HAL, JSON-LD, etc)

API Best Practices: Nouns, CRUD, and More (Part 3)

Reading Time: 11 minutes

This is part three of the API design best practices series.

Once you have an understanding of what your API needs to be able to do in order to meet your developer’s requirements, it’s important to ensure that it remains as flexible and extendable as possible.  Taking advantage of best practices not only means that your API will be familiar to developers, but also ensure that it remains fluid enough to extend and build on top of it in the future.  Here are this week’s best practices to help keep your API agile:

Use Nouns as Resources

If you are building a REST API, be sure that you are using nouns as resources instead of verbs. Verbs are used more with JSON-RPC where you declare the action you are taking in the URL, such as:

/createuser

/edituser/id/1

/deleteuser/id/1

As you can see, verbs are tightly coupled to a specific action and offer very little, if any flexibility.  By using nouns, as recommended with REST and taking advantage of HTTP Action Verbs (such as GET, POST, PUT, etc) you can have loosely coupled resources that are able to accomplish multiple tasks and have new actions added at any time.  Combined with hypermedia, this means that your users would be able to take advantage of any new actions immediately.

Your RESTful endpoints should look like:

/users

/users/id/1

/messages

Keep in mind that it is usually a good idea to use the plural form of the noun unless all available actions are explicit to the singular form.

Use CRUD

CRUD stands for Create, Read, Update, and Delete. But put more simply, in regards to its use in RESTful APIs, CRUD is the standardized use of HTTP Action Verbs. This means that if you want to create a new record you should be using “POST.” If you are trying to read a record, you should be using “GET.” To update a record utilizing “PUT” or “PATCH.” And to delete a record, using “DELETE.”

It’s also important to understand the difference between PUT and PATCH, as PUT is used to perform a complete override of the record (if fields are not included, they are removed) where-as PATCH is used to update a record based on the information provided (or patch it with what you provide), leaving any fields not passed along intact.

Keep in mind there are several different HTTP Action Verbs available, and it’s easy to want to incorporate these new verbs and make your API new and different. However, when it comes to building an API, that’s the last thing you want to do. Utilize CRUD as that is what developers will be looking for when trying to utilize your API, and by deviating from it you are only make your API harder to use and maintain. Also keep in mind that not all browsers or servers support all the HTTP verbs currently out there.

Last but not least, do not mix HTTP Action Verb definitions – if you are telling your developers to make a POST, do not pass the data along as a querystring. That is not the definition of POST and will only cause more confusion for your users.  Make sure you are staying consistent with the verb definition.

Use JSON (when possible)

JSON, or the JavaScript Object Notation format allows for the quick serialization and deserialization of objects. JSON provides a compact format for accessing data, minimalizing the data transfer required while also offering broader language support than XML.

These advantages have made it the format of choice for many developers and the leading format for use within REST APIs. Again, you’ll want to choose the format that is best for your clients, but in the event that you’re encoding objects as XML, I would strongly suggest also offering JSON as an alternative as the serialization is fairly quick and painless.

Keep in mind that one of the advantages to REST is that it is not limited to a single content type, and can return multiple formats if desired.

Use the Content-Type Header

In order to truly keep your API flexible and extendable for the future, it’s also important to build it not just for today’s leading formats, but also for whatever the future may hold.  Until just recently XML was the format of choice for web services, but then JSON came along.  Everyday new formats are being created and used, most notably YAML.  For this reason it is important that your API does not constrain itself to only one format.

It’s perfectly acceptable to only have one format available if that meets your developer’s needs (for example, only taking advantage of JSON), but that doesn’t mean you shouldn’t be planning ahead.  By taking advantage of the Content-Type header you can see what format of data your users are sending you, and in return send back that same format.  For example, if a user sends a request using text/xml, you could send back an XML response while another user could send an application/json request and you could reply with JSON.

By building this functionality into your API to begin with, as new formats become available, or as you add high profile clients with special needs, you are now able to adapt to these new format requests without impacting other users, or other aspects of your API.  This also means that you can stay on top of the technology curve without worrying about having to migrate users from one format to another, or from one API to another.

Ideally, your API’s architecture should be flexible enough to receive content in one format (as described by the Content-Type header) such as XML and return the response in another format based on the Accept header, such as JSON.  Doing so is NOT a good practice, but it does reinforce just how decoupled your API should be from the technology stack, as well as how important incorporating view libraries to handle the output are.  In other words, your API’s architecture should be versatile enough to do this, but it is probably best just to rely on the content-type instead of mixing data formats based on both the content-type and accept headers.

Go to Part 4: Hypermedia →

  • What is Hypermedia
  • Understanding HATEOAS

Be sure to check out previous posts in the series:

Be sure to follow @MuleDev on Twitter to get the latest API Best Practices posts and developer news.

Follow @MuleSoft on TwitterFacebook, and LinkedIn for all the latest in integration.


Tuning Mule ESB: When Performance Matters

Reading Time: 5 minutes

Performance always matters

An application and its environment should be designed with performance in mind. The application, the server it runs on, the database it communicates with, the network it communicates over: all of these elements should be performant. Creating an efficient, elegant mechanism is not only important for a business, but a matter of skill and pride for engineers.

Although true, perhaps that is not the answer one is looking for. When does performance matter? is too broad. The following discussion helps clarify the question, how to answer it, and what to do about it.

Good applications and protocols should work well on a variety of systems. By contrast, performance testing and tuning have system-dependent components. For instance, an application can be tuned to balance throughput versus response time in ways that are mutually exclusive. So out-of-the-box settings may not strike the perfect balance that yields optimal performance for a given use case.

Continue reading

Thanks for connecting with MuleSoft!

Reading Time: 5 minutes

Well, it’s that time of the year again! This is when we put our phones down (well, mostly) and give thanks for everything and everyone around us. We at MuleSoft have a lot and a lot of people to be thankful for. Business is going well (see our Record Q3 Performance announcement), we’re gaining industry recognitions (see our Awards page and in particular, the last Deloitte’s Technology Fast 500 report), and we keep growing. But it’s in times like these that we want to thank everybody who has helped us get where we are today. It’s our customers who believed in our products and always gave us their brutally honest feedback that allowed us to improve and make a better product. It’s all of the Muleys who have poured their energy day-in day-out in building this company and making our vision a reality. But I would like to take a few moments to thank some people who rarely get all the credit they deserve for the work they do: developers.

We provide the tools, but developers are the ones who make something beautiful and amazing with them. Developers are the people who bring to life the magic of MuleSoft platform in the most diverse and challenging environments. They’re the ones who connect the world’s apps, data, and devices to create a sum greater than the parts. When an emergency room is able to access our medical records from the other side of the world, when we look up on our phone when the next bus is arriving, when our plane reservation gets done in seconds, when the green power of a wind turbine is remotely managed… We have to thank the developers who built those connections and make the world a better place for us all.

This is why it is so important to have a vibrant developer community. Some developers go the extra mile and are passionate about advancing the projects they believe in and helping other developers. I am especially thankful for the amazing developer/contributors we’ve met on our journey from the first steps we took with Mule ESB to a fully fledged complete integration platform spanning SOA, SaaS, and API platform. Beyond simply using our tools, these developers help build the very tools themselves. Contributing to the Mule ESB and related projects on GitHub from the very start. This community has provided invaluable contributions to all of our open source projects. This community has kept us honest submitting bugs reports whenever new versions were released. This community is always ready to help fellow developers on MuleSoft community forums, on Stackoverflow, or anywhere there’s a need for a connection. This is why today I want to thank our developer community for all they have done.

While Thanksgiving may be an American tradition, this global community is the heart and soul of MuleSoft. So to our developers we want to say, “Thank you.

Anypoint Platform for APIs – November release

Reading Time: 6 minutes

I am excited to announce that our November release of the Anypoint Platform for APIs is now live. This update to the platform includes a variety of new features and workflow enhancements aimed at streamlining the process of managing APIs. These new features include the following:

  • Improved proxy configuration & support for HTTPS and load balancer scenarios
  • API version deprecation
  • Swagger file import & folder support in API Designer
  • Analytic tracking of policy violations

Proxy Configuration Improvements

As part of the November release of the Anypoint Platform for APIs we have also released a new version of the API Gateway, version 1.3 which you can download here. The new API Gateway includes enhancements that now make it possible to easily deploy API proxies in a loadbalancer environment as well as use a shared port for HTTP and HTTPS endpoint. Shared ports allow you to deploy multiple API proxies to a single gateway. As a result, we’ve modified the proxy generation interface in the platform. Please note, that to take advantage of these new updates to the API proxies you will need to use the latest API Gateway version 1.3. To learn more about configuring and deploying proxies in the Anypoint Platform for APIs you can find full documentation here.

 

Continue reading

API Best Practices: Spec Driven Development (Part 2)

Reading Time: 9 minutes

This is part two of the API design best practices series.

Define Your API in a Flexible, but Standard Spec

I cannot stress the importance of spec driven development enough.  One of the quickest ways to kill your API is to define the API in your code, instead of coding to its definition.  By utilizing an API modeling spec such as RAML you can quickly build out your API in a consistent manner using code and pattern reuse.

Utilizing pattern design and code reuse helps to ensure that your API remains uniform across the full interface, keeping resources and methods alike standardized and easily implemented by your developers.

Tools like API Designer allow you to view your API as it is being designed and watch for inconsistencies/ dependencies you might have missed otherwise.  And perhaps most importantly, once your spec is in place it keeps everyone on the same page, ensuring that your API works exactly the way you want it to.

RAML provides a quick, powerful, semantic, yet human readable format for describing your API.  The API Designer makes it simple to get started and even shows you what your API looks like as you describe it.

 

Mock Your API and get User Feedback

Another huge advantage of tools like RAML or Swagger is that they allow you to mock your API.  This means that you can not only build your API in a visual interface and take advantage of the very same best practices we utilize in development, but you can also share a mock version of your API with potential clients.

Using MuleSoft’s API Designer you can easily turn on a mocking service that gives you a URL that can be shared with other developers.  This allows your clients to “test out” your API by making real calls as if they would from their application.  By utilizing the example responses defined in the RAML file developers can quickly identify issues and inconsistencies, helping you eliminate the majority of design related issues before development even starts.  And by passing along tools like the API Notebook, developers can interact with your Mock API through JavaScript without having to code any calls themselves, and also having the ability to send you the specific use case back giving you true examples of what your developers are trying to accomplish.

The API Notebook handles complex negotiations like OAuth while providing tool tips to walk users through the different API resources and methods, letting them try out and explore your API without server-side coding or having to dig through extensive documentation.

This process can be repeated as necessary, as modifying the spec and creating a new mock only takes minutes, empowering you to perfect the design of your API and ensure that it not only meets your developers’ needs, but also provides a solid, strong foundation for the future of your API.

After all, the nemesis of a long-lived API is not the code, nor the system architecture, but the API design itself.  No matter how careful you are with your API, without a solid foundation it will crumble quickly, costing you thousands to hundreds of thousands of dollars down the road.  It’s better to take the time now, up front and ensure that your API is well designed.

Code to the Spec… and Don’t Deviate

I cannot emphasize the importance of coding to the spec enough.  If you have taken advantage of the above steps and carefully laid out your API, carefully designed your spec, user tested, and perfected your API– there is nothing worse than throwing all that work away by deviating from the spec and doing one-off things.

One of the main reasons for REST was to focus on long-term design, or as Dr. Roy Fielding pointed out, we as humans, as developers are very good at short term design, but horrendous at long-term design.  What may seem like a good solution in the short-term, if not carefully thought out and tested long-term is likely to create big problems down the road.

Think of it like this, how many times have you written code only to look back at it three months later and wonder “what was I thinking?!”  Your API is a contract, and unfortunately the one thing you cannot fix is poor design.

For that reason it’s important to avoid editing your spec during the development cycle.  Instead, if you find an issue with the spec, go back to the design phase where you can visualize the changes, prototype them, and get user feedback.  Then once you are sure your spec provides a solid foundation you can continue with development.

After all, you worked hard to involve your users and create the perfect spec, let’s make sure you use it.

Go to Part 3: Nouns, CRUD, and More →

  • The importance of using Nouns as Resources
  • CRUD and the HTTP Action Verbs
  • Accept & Content-Type Headers
  • JSON vs XML

You can learn more about RAML at RAML.org, or start using MuleSoft’s FREE API Design and Interaction tools with the Anypoint Platform for APIs.

API Best Practices: Plan Your API (Part 1)

Reading Time: 13 minutes

This is part one of the API design best practices series.

Understand WHY you are building an API

Perhaps the foundation of the foundation, understanding why you are building an API is a crucial step towards understanding what data/ methods your API should make accessible and how your users will utilize it. Unfortunately, API is a buzzword right now, and many companies are rushing to build APIs with the idea that “we’re going to make our data accessible and our users will love it!” There’s probably some truth to that, but that is not a good enough reason. What exactly are you making accessible and why? Who are your API users – are they your customers, or third party services, or developers who are looking to extend upon your application for their customers? Understanding the market you are serving is vital to the success of any product or service.

Key questions to ask:

Continue reading

New Series: API Design Best Practices

Reading Time: 5 minutes

By now, you’ve probably already seen the image of the iceberg cross section showing just how many APIs are available out in the world. With over 13,000 public APIs available for use across the web, and hundreds of thousands more being used privately and in-house, the possibilities are endless.

The demand for flexibility and extensibility has driven the development of APIs and tools alike, and in many regards it has never been easier to create an API than it is today with multitudes of frameworks (such as JAX-RS, Apigility, Django REST Framework, Grape), specs (RAML, Swagger, API Blueprint, IO Docs), and tools (API Designer, API Science, APImatic) available.

However, despite the predictability of the demand for APIs, this tidal wave has taken many by surprise. And while many of these tools are designed to encourage best practices, API design seems to be constantly overlooked for development efficiency. The problem is, however, that while this lack of focus on best practices provides for a rapid development framework, it is nothing more than building a house without a solid foundation. No matter how quickly you build the house, or how nice it looks, without a solid foundation it is just a matter of time before the house crumbles to the ground, costing you more time, energy, and resources then it would have to simply build it right the first time.

Continue reading

Performance Impact of an IO-Intensive Application

Reading Time: 11 minutes

If you’ve ever worked on performance issues with an IO- intensive app, there is a good chance you already know that the application performance degrades when the disks are stressed out. This fact is usually well known, but the reasons behind it aren’t always clear. I’d like to try and clarify what’s going on behind the scenes.

In a typical scenario, when data is written to a file, it is first written to a memory area reserved as the page cache. The page holding the newly written data is considered dirty. After a period of time per the kernel IO policy, the kernel flushes the dirty data to the device queue to persist to hard disk. Once the data gets to the queue, the rest is mechanical: The device driver reads the IO requests, then spins, seeks and writes to the physical disk block where the file is. The journal file is written first if enabled, then the actual file.

In a recent discussion with a few other engineers, an idea of disabling file system journaling came up to improve disk write latency. While this is certainly true because it is one less disk operation per write, the actual time gained is negligible because the journal file is in the same block as the file to be written. The benefits of having the journal file to restore the disk from a crash far outweighs the little latency saved.

Continue reading