...
IS&T web applications contain complex browser-based user interfaces created with JavaScript, HTML and CSS. The functional/regression test tool we currently use, QTP, only works in Internet Explorer, so it does not test IS&T’s set of supported web browsers. Therefore QTP cannot test individual browser issues. Also, IS&T needs to evaluate its web applications against new web browsers and changes to the IS&T web application infrastructure (new database, new application server, new VM, etc.)
What we did:
- We convened a committee with representation from the Help Desk, the Quality Assurance Team, Web Services, DCAD and Student Systems, and began studying the current web application testing landscape.
- We looked at a list of approximately 82 different automated Test Tools (http://www.softwareqatest.com/qatweb1.html#FUNC) and choose 6 worthy of further evaluation, based on the following criteria:
- Support for all MIT operating systems
- Support for all MIT browsers
- Ability to playback tests in a browser (instead of testing via browser emulation)
- Ability to record tests (to make test creation easier)
- Some sort of gui (versus code only framework)
- Decent documentation
- Some sort of name in the industry
- We created a test plan to evaluate these Test Tools (the test plan is here). The test plan was heavily weighted with DOM manipulation, AJAX and other JavaScript functionality. (expand)
- We broke up into groups of 1 to 3 users to try each of the 6 Test Tools.
- We modified the Test Plan as needed.
- We found the following:
criteria | AppPerfect | EggPlant | FuncUnit | QTP 11 | Selenium | Squish |
---|---|---|---|---|---|---|
adequate documentation (1 to 10 scale) | 3 | -2 | 5 | 6.5 | 6.5 | 4 |
longevity/viability of vendor (1 to 10 scale) |
| 8/unknown | 5yrs/ | 20yrs/10 | 9yrs/9 | unknown |
recorded test plan | yes | yes | no | yes | yes | yes |
ran test plan in IE 9/Win | no | yes | did not test because it could not record | yes | yes |
|
ran test plan in Firefox 8/Win | no | yes | did not test because it could not record | no | yes |
|
ran test plan in Firefox 8/Mac | no | did not test | did not test because it could not record | no | yes |
|
ran test plan in Safari 5/Mac | no | did not test | did not test because it could not record | no | no | sort of |
Conclusions |
|
|
|
|
|
|
...