Final Report

Objective:

To evaluate functional web application testing tools to better meet IS&T's multi-browser,multi-platform testing needs.

Scope:

IS&T web applications include many complex browser-based user interfaces created with JavaScript, HTML and CSS.  The functional/regression test tool we currently use, QTP, only works in Windows, so it does not test IS&T’s set of supported web browsers. Therefore QTP cannot test individual browser issues.  IS&T needs to evaluate it's web applications against new web browsers and changes to the IS&T web application infrastructure (new database, new application server, new VM, etc.). Therefore we convened a cross-area team to evaluate functional web testing tools to see if any would fit IS&T's needs.

What we did:

1. We convened a committee with representation from the Help Desk (Lisa Robinson), the Quality Assurance Team (Sean Velury and Don Flanders), Web Services (Michael Berger and Ed Orsini), Stellar (Judith McJohn) Student Systems (Felicia Young), and SWRT (Alex Kozlov) and began studying the current web application testing landscape.
2. We looked at a list of approximately 82 different automated Test Tools (http://www.softwareqatest.com/qatweb1.html#FUNC) and choose 6 worthy of further evaluation, based on the following criteria:
   1. Support for all MIT operating systems
   2. Support for all MIT browsers
   3. Ability to playback tests in a browser (instead of testing via browser emulation)
   4. Ability to record tests (to make test creation easier)
   5. Some sort of gui (versus a code only framework)
   6. Decent documentation
   7. The team's subjective evaluation of the company's reputation and ability to stay viable
3. We created a test plan to evaluate these Test Tools (view the test plan here). We created the test plan based on an existing administrative web application called APR Hires. We chose APR Hires because it used many different kinds of JavaScript and AJAX functionality. The test plan involved data input of text, numbers, dates and dollar amounts. It included testing AJAX calls, JavaScript Validation of user input, and insertion of new DOM elements. The test plan contains positive test assertions (checks in the software to confirm that the expected result occurred) for most user input. The test plan did not contain negative assertions (tests that unexpected things did not happen).
4. We broke up into groups of 1 to 3 users to try each of the 6 Test Tools.
5. We modified the Test Plan as needed.
6. We found the following:

1. eggPlant was image based. It required the testers to laboriously create and update images for every element that was being tested. It was very fragile, as different views of the same page required more taking more screen grabs. We determined that the cost and frustration of upkeep made the product very problematic.

2. Squish has basic functionality, but advanced testing required the tester to build Javascript functions. As this is an expensive product we chose to stop testing at this point in favor of Selenium.

Lessons Learned

Record and play is a myth; it turned out that programming was needed with every tool to be able to reproduce the test case.

Comprehensive tests require if/else programming, looping, assertions and negative cases at the very least. Simply recording, sprinkling simple assertions and playing back does not create a comprehensive test.

Documentation for automated Test Tools is underwhelming.

What We Recommend:

We recommend rolling out Selenium to IS&T web developers because it met most of our requirements, with the notable exception of Safari 5 support:

Caveat

1. Selelenium does not support the current version of Safari (5) that ships with OSX 10.6 and 10.7. See http://code.google.com/p/selenium/issues/detail?id=573.

Success Factors for rolling out Selenium

1. We will need the following dedicated hardware set up to run Selenium tests for OSX 10.6 with Firefox, 0SX 10.7 with Firefox, Windows 7 with IE and Firefox, and Linux with Firefox.
2. In order to be successful, IS&T will need to task someone at least part time to be the Selenium product evangelist. This person will create documentation, build some baseline tests, handle problem items (type versus type keys), and trains developers and help desk how to use Selenium, how to create proper tests and how to run their tests on the IS&T hardware.
3. We will need to create test users in the various systems who have Touchstone and web certificates.
4. We will need someone tasked with coordinating and maintaining the testing hardware and software. This could be the Selenium evangelist or someone else.
5. In order for this project to be successful, it will need management buyin and continued support to create and maintain the above.
6. To make this happen, we would need to create a project with a project plan and project manager.

Conclusions:

Automated Test Tools, in general, give the most 'bang for the buck' when used for regression testing. Due to the constantly changing browser/platform playing field no test tool can be expected to support the latest browser/platform combination. Therefore we conclude that automated Test Tools cannot be the sole method for testing MIT-supported applications.

The current browser space is expanding and changing and churning rapidly right now, making it even harder for Test Tools to catch up.

Also user and management expectations around browser support and platform are constantly expanding. Chrome, mobile in general and iPhone are the current issues, Windows 8 is looming on the horizon.

Therefore supporting constant rollout of the latest browsers requires manual testing resources. However, automated Testing Tools do improve the quantity and coverage and reduce time for testing Test Tool-supported browsers.

Further Study:

Once QTP 11 final ships, retest the Test Plan in QTP 11 for both IE 9 and Firefox.

What are other large institutions doing?