I did Unit Testing, but the Customer is still not happy
I have this web project. Pretty complex one. The code coverage is over 90%. Awesome! I even have a Continuous Integration set up. My code is built, and my tests make sure that functionally works as expected. I even have a well defined suite of UI tests using an internal renderer that is super fast. But, ohhh… today a Customer filled in a complaint saying that the site is not working well. Buttons are not responding, images are missing and the like. Catastrophe! This may be familiar to you? What am I missing here? I need external, not controlled-by-us browser behavior testing! How do I solve this problem? Or at least have an early warning that Sauron is about to be awake?
UI Test Automation is the Key
There are many available tools for web testing, all of them have advantages and disadvantages. The key concepts to take into account are re-usability, maintainability, and ability to reproduce as close as possible the actual user experience in a programmatic way (happy user → happy Customer → Christmas bonus!) .
Tools based on graphics/coordinates?
You have a screen, you define some actions based on coordinates on the screen and you develop your tests accordingly. Even better, the tool provides you with a UI to make your life simpler. Everyone is happy until the web designer has this crazy idea about improving usability and change the order of Save – Cancel buttons, to Cancel – Save. Tests start behaving in a strange way. The problem is not easy to debug. And we are not talking of a major UI design change.
There is a big improvement to this approach: Instead of coordinates, small screenshots are used to identify a certain element within a page (eg Project Sikuli). It is definitively a major strategy improvement, but you will have the same problems as when using coordinates if changes in the UI are not trivial ones.
The Bad: Not very scalable, low re-usability, poor maintenance.
The Good: Quick solution. Good for proof of concept.
What about Native Windows testing tools?
You have access to the internals of the applications under testing (data items, controls, widgets, etc.) . Of the tool that I used that take this approach (i.e. TestComplete ), I can say that it is a paid solution per workstation; it was not portable to other OS (making difficult the integration with an existing Continuous Build machine) and has many re-usability problems. Last, but not least, element identification was done mostly using the actual text displayed on-screen. Which is bad… Think for one second about tests defined that way and internationalization… Ufff!
The Bad: Not very scalable, low re-usability, poor maintenance, possible i18n issues, possible poor integration with CI solutions, portability issues, and you are not using the actual browser to test.
The Good: Access to the internals of the application. Easy to do assertions over third party artifacts (eg. contracts generated in PDF format by the system)
What about test recorders?
In my opinion, test recorders such as Selenium IDE are really a bad solution for medium/large projects. Let’s say that you record 100 tests. Then, something changes in your web project. If it is a core change you will have to work on most of your hundred tests to make them run again. On the other side you do have Page Models (they are developed not automatically generated), where you make a change in one or two classes and automatically all your hundred tests start working like a charm. I’ll talk about this next.
Tools that use Javascript injection
This is the approach we finally decided to use. But which tool to use? What about Selenium RC? What about Windmill? This last one looks pretty good at a first glance… And at a second one too… But we finally choose Selenium. Why?
Selenium versus Windmill
- Re-usability and maintainability much better in Java than in Javascript if you apply the correct patterns. For example, decoupling accesor logic from actual tests (Page Model Pattern).
- With Selenium you can use the same IDE in which you develop your code to debug your tests too, increasing developer productivity.
- Selenium can be integrated with existing testing frameworks, for example JUnit and TestNG, while Windmill has its own test runner. So in existing projects you cannot take advantage of existing configurations, for example in your CI machine.
- Easy to set preconditions for the tests. Scenario: Before testing, a couple of newly created war files should be deployed into tomcat, and after the tests are run, undeployed. How you do that with a pure Javascript solution? With Maven and Cargo is easily done.
- Selenium is working hard on Selenium 2.0, which will solve the problems with IE among others.
- Selenium injects into the page the minimal amount of code necessary to run the tests, while with Windmill all the tests have access to the context of the page. Heisenberg’s Uncertainty Principle says that the Observer affects the Measurement. Now, replace “Observer” by “injected Javascript code”, and “Measurement” by “web page to test”… Less “Observer” is less intereference.
Going with Selenium RC
Selenium uses the Server-client paradigm. In what way? The server is like a proxy. Client requests to the server for a certain page, server retrieves the page, injects it with some Javascript code (taking care of not modifying the original HTML code) and sends the page to the client. Because of the injected code, the client will have the ability to simulate user interaction on the page quite close to the way an actual user does it. There are two main Selenium flavours: Selenium-IDE (test recorder/Firefox plugin) and Selenium-RC (to be integrated with Java/C++, etc.). The second is the Choosen One.
The Bad: Yeah, I know. Selenium is not very fast to run. Tests are not easy to develop. It has problems with third party software (for example, assertions over a PDF generated by our system are not supported), and does not work really well with the infamous Internet Explorer (not really a Selenium fault, by the way, but because among other things of the poor support of Javascript, XPath, etc.). So why? Why?!!
The Good: Selenium is open source with a most active community behind it. It can be easily integrated into Java code. Using Java and Maven, you can start up a Selenium Server, a web app server, deploy the application in that server doing a clean deploy, test it using one or more browsers and, if you are in the mood, making a screenshot if a problem is detected. Your solution will be cross-platform too. What about re-usability and maintainability? Here it comes one of the most interesting things about this approach.
Because Selenium can be used in a programmatic way (with Java, for example), you can implement the Page Model Pattern that will save you of several developer hours fixing tests. I will follow up with a post about the Page Model Pattern next.
