Goals

These goals are meant to be high level directions for where we want to go with our work in test development for the next quarter. Below I'll list each high level goal and a set of tactics that we will be using to make that goal a reality.

High Level Goals

• [DONE] [Test Automation] Improve and expand our testing infrastructure by increasing reliability, investigating third party test suites and methodologies.
• [DONE] [Test Development] Expand test coverage for the architectural changes in 1.9.2 as they coalesce and land
• [DONE] [Community] Improve our ability to crowd source QA

We're going to do this in the light of the infrastructure projects we have here: http://spreadsheets.google.com/ccc?key=0AkFMXPiC6OptcjhTZlFST2dyaUxXVlpQeGhFQzV6c2c&hl=en

Which will be tracked here: https://wiki.mozilla.org/QA/TDAI/WorkWeek_Planning

What is below are our notes for future projects and other sprints that might also go into the above list. The google doc is our primary planning and scheduling interface and is the definitive list of individual goals.

Test Automation

Goal:Improve and expand our testing infrastructure by increasing reliability, investigating third party test suites and methodologies.

Tactics:

• [DONE] (Plan to integrate with jsreftest once it lands) Investigate ecma 5 test suite from other vendors (bc)
• [MISSED] JsReftest completed, not integrated with tinderbox Integrate JsReftest with tinderbox (bc)
• [DONE] Create a mechanism for defining and tracking expected failures for fennec (jmaher)
• [MISSED] (Began some of this, did not complete investigating all failures due to testing for releases and getting automatic log compare in place) Ensure all failed tests on fennec are being addressed (jmaher)
• [DONE] Create better test reporting for Fennec using results server (jmaher)
• [DONE] Complete integration of Mozmill with tinderbox (ctalbert & mikeal)
• [DONE] Investigate and create test framework for electrolysis (jgriffin)
• [DONE] Phase 1 completed Automate analysis of topcrash URLs using Sisyphus and Lithium and security analysis tools to help prioritize crash work (aaronMT)

Test Development

Goal:Expand test coverage for the architectural changes in 1.9.2 as they coalesce and land

• [MISSED] (We did this, but not to the extent imagined across the entire team) Work on creating more visibility around test development work by blogging, participating in test dev days and test dev evangelism to developers (all)
• [DONE] Extend FennecMark performance testing (jmaher)
• [DONE] Expand test coverage for animation frameworks that are landing in 1.9.2 (SMIL, Canvas animation, JS animation, CSS Transitions) (mw22)
• [DONE] (XBL2 is ongoing) Expand test coverage for new markup (HTML5, XBL2) (harthur & jgriffin)
• [DONE] Complete DocShell test framework (jgriffin)
• [DONE] (Compositor landed as manual testing) Video backend, Compositor testing (mikeal)
• [DROPPED] (Offered to work with dev on this, but they solved the problem themselves) Front-end performance measurment (mikeal (possibly))

Community

Goal:Improve our ability to crowd source QA Tactics:

• [DONE] Drive QAC to completion (harthur, aaronmt, onno, ctalbert)
  • [DONE] Create a mobile QAC (all of the above + jmaher)
• [MISSED] Drive test dev days (ctalbert)
• [DONE] Evangelize the results server as it goes live (mikeal)

jmaher

• [DONE] extending fennecmark with more tests. Would like to identify and implement 2-3 more tests
• [DONE] helping (develop, port, test, evangelize) qac for mobile
• [DROPPED] symbian - initial prototype of unittests running
• [DONE] provide tools, scripts and support to build in a timely manner as needed to get the wince unittests running on tinderbox
• [MISSED] (Did some of this, will be ongoing) Investigate all failed mobile test and results
• [DONE] see if we can get a more granular results viewer instead of just a knownfailcount as we are currently using (hopefully using the results server)
• [DONE] Push harder to get the failing tests addressed (file bugs, get feedback from component owners, ask for fix or provide patch)
• [DONE] more release testing for Maemo beta and final and Wince alpha and beta releases.

bc

• [DONE] (got the first pass at infrastructure up this quarter, not completed) automate sisyphus to analyze crashreporter urls to repro crashes, save state and repro crash, attempt automatic site reduction and expose those sites in a log
• [DONE] investigate ecma 5 test suite from third party folks (like ms)
• [MISSED] (Got the jsreftest up and coded but not reporting to tinderbox) complete jsreftest and get them reporting to tinderboxen
• [DROPPED] (Turned out not to be needed) help people write mozmill tests by helping teach the javascript they need to know.

mw22

• [DONE] smil & js animation api

harthur

• [DONE] (jgriffin actually did this) HTML5
• [DONE] (Have begun test harness and planning work, will be continuing in next quarter) XBL2
• [DONE] QAC

jgriffin

• [DONE] finish docshell
• [DONE] work on html5 parsing
• [DROPPED] (Dropped in favor of working on electrolysis testing) investigate other test systems, start with mozmill

aaron train

• [DONE] work on the sisyphus automation
• [DONE] QAC

mikeal

• [DONE] stability fixes on mozmill 1.2.x as needed
• [DROPPED] (Dropped in favor of working on brasstacks result viewing infrastructure which enabled a lot of progress on all other test dev projects) writing tests for 1.2.x for various front end UI and platform as appropriate (video and compositor perhaps).