The Zen of UI Testing with Selenium, Hudson and Sauce Labs

Most people who write UIs don’t care about testing. You know why? Because it’s hard. So hard, they’d rather not even bother and test things manually. You have multiple browsers. You have multiple platforms. And worse, you have all these frameworks and toolkits which are difficult to test. I’ll pick on GWT here for a moment. It takes 20 seconds to start a test – let alone a server side component to interact with or the time it takes to run your test. [1]

as an afterthought doesn’t work for SaaS/PaaS offerings though. You’re deploying multiple times a week or even day. You need to be sure that you’re not breaking anything. And there is only one way to do this: tests! And I mean real, actual-in-the-browser-dealing-with-crazy-cross-browser-bugs tests.

There is only one real way to do this as far as I know, Selenium [2][3].  And while it will work with any webapp, it shines when your UI is completely browser based. This is the approach we took with iON – there is no server side framework. It’s a rich Javascript, MVC(ish) UI made with Backbone.js and jQuery, and powered by a set of RESTful services. Having used many UI frameworks over the years this has to be by far my favorite UI setup and the easiest to test that I’ve worked with. And, it’s fast – I can run tests quicker than I can test things manually. This puts proper incentives in places to write tests.

We combine this setup with Sauce Labs support to run tests on their grid against our build server. There is no way I would’ve been able to get things QA’d and working in IE with such little time if I could not have used their infrastructure to do so. QA’ing manually simply takes too long and to set up this infrastructure ourselves would’ve taken forever.

Sidenote: We’re looking for talented Web UI/Javascript engineers! If building great user interfaces/experiences is important to you and technologies like Backbone.js or Selenium are something like you to hack on, let us know.

Here’s how we do it:

  • Our REST services are built with JAX-RS. Our JAX-RS resources are thin shims over our internal services.
  • Internal services have mock backends so that we can easily use fake data and check interactions between the frontend and backend.
  • We have a JUnit class which fires up a Jetty server and runs the Selenium tests
  • For all our browser tests, we execute them against different browsers  (IE7, IE8, IE9, Firefox 4, and Firefox 5 currently) using Maven and Sauce Labs.

Let’s take a look at the details.  First, we need to set up the Selenium driver. We support doing this locally and remotely via Sauce via system properties:

This code gets run before/after each test.  As you can see, by default we use the local FireFox driver. But also allows us to have a simple profile in our maven build which allows us to activate SauceLabs functionality:

Running these tests is dependent on a tunnel between where your tests are running and SauceLabs which allows their browser to talk to your web service. This is a pain to set up via Maven, but luckily there is a nice Hudson/Jenkins plugin! Following the instructions was a snap.

Finally, we needed to get our build to run against all the other browser combinations we wanted. We set up a separate build module for this which takes the tests from our other module and runs them against Sauce.  To do this, ensure your publishing your test jar to your Maven repo by adding this config:

Then depend on the test jar (you’ll also have to add any test dependencies you have since test dependencies aren’t transitive).

And finally, run your tests against different browsers by configuring different executions of the Maven surefire plugin:

Now you’ve got tests running against lots of popular browser combinations with very little, if any infrastructure effort!

Future Improvements

This has been a good step forward for us, but I can point to a few areas I’d like to improve:

  1. I would like to be able to run our tests in parallel. We’d then be able to test our system in a few short minutes instead of 30-40.
  2. Perhaps we should consider running tests via a test runner instead of @Before/@After methods
  3. Coverage – we got this set up late in the game, and I think we could improve a lot on our coverage.

1. If you’re using GWT, check out gwt-test-utils.  Gaël et al have done some awesome stuff to make GWT tests run in a short amount of time. In short they replace every JavaScript JNI method with a Java equivalent on the fly. Tedious work, but he is able to run 1000s of tests in a couple minutes, although not against an actual browser.

2. Jasmine is cool and helpful too, but  it doesn’t deal with cross-browser issues. But it definitely has a place in the JS engineer’s toolkit.

3. I will say that the Sencha folks have done a pretty great job of making their UIs work cross platform without you having to think to much though.

 


We'd love to hear your opinion on this post


4 Responses to “The Zen of UI Testing with Selenium, Hudson and Sauce Labs”

  1. […] Read the rest of this post on the MuleSoft Blog […]

  2. hi Dan,

    how do you quantify and then track testing coverage?

    I have been looking at test management (e.g. http://www.xqual.com) and am interested in exploring the integration possibilities with a test mgmt tool, selenium, hudson etc

    enjoyed the article – like the approach of using mockups for your services

    thanks!

    • We have been using Sonar off and on, but this doesn’t really track code on the frontend. I’m not aware of any tool that will do this for Javascript in the browser. Do you know of any?

  3. […] The Zen of UI Testing with Selenium, Hudson and Sauce Labs illustrates a nice switch in the @Before method to run either locally or in the cloud (in this case with Sauce Labs). Limiting your framework to be always in the cloud or always behind your firewall is silly these days. […]