...
- We created a test plan to evaluate these Test Tools. The test plan was heavily weighted with DOM manipulation, AJAX and other JavaScript functionality. (expand)
- We broke up into groups of 1 to 3 users to try each of the 6 Test Tools.
- We modified the Test Plan as needed.
- We compared notes and removed EggPlant, AppPerfect and FuncUnit because:
- AppPerfect: Could not parse an html select and we were not able to get support from the vendor, (as Ed for details)
- EggPlant: Because it is image based, it does not directly test JavaScript and it was maintenance intensive.
- FuncUnit would not record in newer versions of FireFox than 3.X (confirm) and otherwise was similar to and based on Selenium
Lessons Learned
Record and play is a myth; it turned out that programming was needed with Every tool to be able to reproduce the test case.
Comprehensive tests require if/else programming, looping, assertions and negative cases at the very least. Simply recording, sprinkling simple assertions and playing back is NOT a comprehensive test.
Conclusions:
Automation Test Tools are designed for regression testing against existing functionality. They are not designed to test future functionality or future browsers.
...