...
criteria | AppPerfect | EggPlant | FuncUnit | QTP 11 | Selenium | Squish | ||
|---|---|---|---|---|---|---|---|---|
adequate documentation (1 to 10 scale) | 3 | -2 1 | 5 | 6.5 | 6.5 | 4 | ||
longevity longevit in yearsy/viability of vendor (vendorin 1 to 10 scale ) | unknown/unknown | 8/unknown | 5yrs/unknown | 20yrs/10 | 9yrs/9 | unknown/unknown | ||
recorded test plan | yes | yes | no | yes | yes | yes, but test would not replay without JavaScript coding of test. 2 | ||
ran test plan in IE 9/Win | no | yes | did not test because it could not record | yes | yes did not test | no, see above. 2 | ||
ran test plan in Firefox 8/Win | no | yes | did not test because it could not record | no | yes |
| no, see above. 2 | did not test |
ran test plan in Firefox 8/Mac | no | did chose not to test. 1 | did not test because it could not record | no | yes |
| no, see above. 2 | did not test |
ran test plan in Safari 5/Mac | no | did chose not to test. 1 | did not test because it could not record | no | no |
| no, see above. 2 |
|
1. eggPlant was image based. It required the testers to laboriously create and update images for every element that was being tested. It was very fragile, as different views of the same page required more taking more screen grabs. We determined that the cost and frustration of upkeep made the product very problematic.
2. Squish has basic functionality, but advanced testing required the tester to build Javascript functions. As this is an expensive product we chose to stop testing at this point in favor of Selenium.
Lessons Learned
Record and play is a myth; it turned out that programming was needed with every tool to be able to reproduce the test case.
...
Documentation for automated Test Tools is underwhelming.
What We Recommend:
Roll We recommend rolling out Selenium to IS&T web developers with the following requirements and caveats:because it met most of our requirements, with the notable exception of Safari 5 support:
Caveat
- Selelenium does not support the current version of Safari (5) that ships with OSX 10.6 and 10.7. See http://code.google.com/p/selenium/issues/detail?id=573.
Success Factors for rolling out Selenium
- We will need the following dedicated hardware set up to run Selenium tests for OSX 10.6 with Firefox, 0SX 10.7 with Firefox, Windows 7 with IE and Firefox, and Linux with Firefox.
- In order to be successful, IS&T will need to task someone at least part time to be the Selenium product evangelist. This person will create documentation, build some baseline tests, handle problem items (type versus type keys), and trains developers and help desk how to use Selenium, how to create proper tests and how to run their tests on the IS&T hardware.
- We will need to create test users in the various systems who have Touchstone and web certificates.
- We will need someone tasked with coordinating and maintaining the testing hardware and software. This could be the Selenium evangelist or someone else.
- In order for this project to be successful, it will need management buyin and continued support to create and maintain the above.
- To make this happen, we would need to create a project with a project plan and project manager.
...