Compatibility/Automated compat testing
Contents
About
The tool Compatipede (built on SlimerJS for Gecko and PhantomJS for Webkit rendering) can do automated testing of live websites.
The framework consists of the projects jannah for the infrastructure that's capable of scaling up to testing many pages at once across many computers and boar for the script that controls SlimerJS/PhantomJS.
Goals
Using collected data, we will be able to
- Automate some of the analysis of broken sites - we can query the database for logged information and screenshots, and suggest likely reasons a site is broken
- Track broken sites, see if they get fixed
- Track fixed sites, see if they regress
- Discover problems we're not yet aware of
Roadmap
Compatipede 2 is still under development. This is a prioritised list of tasks we would like to complete:
- Implement support for plugins [1] - specific snippets of JS to check for specific problems
- Database for result storage (compat-entomology re-use or similar schema)
- Integrate with DB for result storage
- Generating reports from DB contents
- Improve client scripts - implement scaling up to n tabs functionality
- Automated tests for the framework
- Support getting URLs directly from Bugzilla or Webcompat searches for a test run (likely in a client script)
- Support AWCY list mode - running all sites on an AWCY list (likely in client script)
- Do live CSS analysis across Gecko+WebKit, find a suitable way to store this information
- Track and report JS errors (preferably flagging those that happen in one of the engines but not the other)
- Track and report redirects
- Way to run Hallvord's existing regression tests from the old extension / slimertester.js
- Selenium / WebDriver support to facilitate easier automation, e.g. log in
Re-purposing Compatipede 2
The above features will be excellent for web compatibility testing. However, the tool we write can also be used in various other ways, including
- Stability testing - loading many sites w/o crashing
- Browser regression testing - comparing output from Gecko version n and Gecko version n+1
- Memory analysis - using Gecko's built-in methods for dumping memory stats or OS reporting
- A special case of memory analysis might be to compare memory usage for Gecko and Webkit when loading the same page, might give us hints about structures and sites Webkit handles better than Gecko memory-wise
Compatipede 2 architecture
Test VMs
We currently have two machines allocated to us for prototyping:
- compatipede1.dmz.scl3.mozilla.com
- compatipede2.dmz.scl3.mozilla.com
Points of contact are first conor@forwind.net and then miket@mozilla.com.
