Scope:
IS&T web applications contain complex browser-based user interfaces created with JavaScript, HTML and CSS. The functional/regression test tool we currently use, QTP, only works in Internet Explorer, so it does not test IS&T’s set of supported web browsers. Therefore QTP cannot test individual browser issues. Also, IS&T needs to evaluate its web applications against new web browsers and changes to the IS&T web application infrastructure (new database, new application server, new VM, etc.)
What we did:
- We convened a committee with representation from xxxxxxx, and began studying the current web application testing landscape.
- We looked at a list of X amount of Test Tools (http://www.softwareqatest.com/qatweb1.html#FUNC) and choose 5 worthy of further evaluation, based on the following criteria:
-
- Support for all MIT operating systems
- Support for all MIT browsers
- Ability to playback tests in a browser (instead of testing via browser emulation)
- Ability to record tests (to make test creation easier)
- Some sort of gui (versus code only framework)
- Decent documentation
- Some sort of name in the industry
- We created a test plan to evaluate these Test Tools. The test plan was heavily weighted with DOM manipulation, AJAX and other JavaScript functionality. (expand)
- We broke up into groups of 1 to 3 users to try each of the 6 Test Tools.
- We modified the Test Plan as needed.
- We compared notes and removed EggPlant, AppPerfect and FuncUnit because:
- AppPerfect: Could not parse an html select and we were not able to get support from the vendor, (as Ed for details)
- EggPlant: Because it is image based, it does not directly test JavaScript and it was maintenance intensive.
- FuncUnit would not record in newer versions of FireFox than 3.X (confirm) and otherwise was similar to and based on Selenium
-
criteria
AppPerfect
EggPlant
FuncUnit
QTP 11
Selenium
Squish
adequate documentation (1 to 10 scale)
3
-2
5
6.5
6.5
4
longevity/viability of vendor
8/unknown
5yrs/
20yrs/10
9yrs/9
unknown
recorded test plan
yes
yes
no
yes
yes
yes
ran test plan in IE 9/Win
no
yes
did not test because it could not record
yes
yes
ran test plan in Firefox 8/Win
no
yes
did not test because it could not record
no
yes
ran test plan in Firefox 8/Mac
no
did not test
did not test because it could not record
no
yes
ran test plan in Safari 5/Mac
no
did not test
did not test because it could not record
no
no
sort of
Conclusions
Lessons Learned
Record and play is a myth; it turned out that programming was needed with Every tool to be able to reproduce the test case.
Comprehensive tests require if/else programming, looping, assertions and negative cases at the very least. Simply recording, sprinkling simple assertions and playing back is NOT a comprehensive test.
Documentation for Test Tools is underwhelming.
Conclusions:
Automation Test Tools are designed for regression testing against existing functionality. They are not designed to test future functionality or future browsers.
Test Tools, by their nature, are one step behind the rapidly changing current browser versions. Therefore Test Tools will never be the ONLY testing solution if MIT is going to support current browsers.
The current browser space is expanding and changing and churning rapidly right now, making it even harder for Test Tools to catch up.
Also user and management expectations around browser support and platform are constantly expanding. Chrome, mobile in general and iPhone are the current issues, Windows 8 is looming on the horizon.
Therefore supporting constant rollout of new the latest browsers requires manual testing resources.
However, automated Testing Tools do improve the quantity and coverage and time for testing Test Tool-supported browsers.