QA/Goals/2015q1

From MozillaWiki
< QA‎ | Goals
Jump to: navigation, search

Milestones this Quarter

This is the list of milestones this quarter. The table below is used to judge how our efforts shaped each axis of quality in the final release.

Milestone Date Deliberate Impact Mozfactor Overall Quality Rationale
Firefox 35 Jan 12 2015
Firefox 36 Feb 23 2015
Spring Release Planning Ongoing
New Embedded Team Sprints ongoing
Marionette Training and Test Conversion Feb 1 2015

Prioritized Tasks

General Quality

  • [DEFER] [whimboo, with help: ctalbert, kairo, releng] Automate kicking off update tests so that the release process is automated end to end except manual spot checks
    • Defered due to time being spent on Mozmill -> Marionette Conversion and the possibility of doing update tests in buildbot. Will be coordinating with Releng
  • [DONE] [stephend, joint with a*team] Coordinate training and prioritize efforts to convert P1 tests from Mozmill to the Marionette "Green-light" test framework in preparation for E10s
  • [ON TRACK] [whimboo, a*team] Get Marionette Green-light tests running in the QA CI system for at least Nightly tests
  • [ON TRACK] [mschifer] Create and monitor a set of risk factor metrics to understand risks in areas where we have no embedded QA
  • [ON TRACK] [kairo] Create a better concept for tracking crash-stats that more reflects the actual risk and severity of the crashes involved (than the current top 10 list)

Firefox

E10s

  • [DONE] [lizzard, juanb] - Establish baselines for automated-test coverage, crash-stats, add-on compatibility, and memory footprint for e10s Nightly and track them throughout Q1.
    • We did this for addons, automated tests, and crash stats. Memory footprint did not seem like a reliable metric to track at this time given the amount of work required and the amount of variability in the data.
  • [AT RISK] [lizzard, juanb] - Improve the automated-test coverage for E10s-enabled builds by turning on tests for a new platform and ensuring bugs filed for perma-fails.
    • This is bug 1122901. It is now in the hands of the automation team and the releng team and while we are pushing on it, it's not entirely in our control anymore.
  • [ON TRACK] [lizzard, juanb] - Engage community to help improve add-on compatibility testing for e10s

DOM

  • [DONE] [ashughes] - Document best practices for triage and test writing for DOM issues to pave the way for community involvement
  • [DONE] [ashughes] - Establish baseline metrics for specification coverage, test stability, and other DOM quality metrics and track them throughout Q1
  • [AT RISK] [ashughes] - Create several new automated tests/get existing Web-platform tests working to support the Picture tag and ServiceWorkers specifications
    • this is at risk due to re-org and higher priority deliverables being added (eg. Betabreakers)

Security/Privacy

  • [ON TRACK] [mwobensmith] - Set up AWS instance to run large TLS regression test
  • [ON TRACK] [mwobensmith] - Create a fast-running subset of the larger TLS test and automate to run per-release (i.e. pre-beta)
  • [ON TRACK] [kjoziwak] - Monitor and create telemetry probes to gather data to help serve as an early warning system for risky changes

Fennec

  • [ON TRACK] [aaronmt, kbrosnan] - Slim down the per-release smoketest by at least 25%
  • [ON TRACK] [kbrosnan] - Investigate using Marionette/Firefox Green-light on Fennec, create a plan and prioritized set of targets for the Marionette team
  • [ON TRACK] [aaronmt] - Fix or discontinue appthwak automation

Web QA

Platform QA

  • [ON TRACK] [Marcia, Syd] Eliminate Flash issues on YouTube by supporting the media team in shipping MSE on YouTube for Windows Vista and higher through manual and automated testing

Prioritized Tasks

  • [DONE] [Maja] Create and deploy a suite of automated YouTube playback tests.
  • [DONE] [Maja] Write a tool to pull a subset of YouTube URLs from crash-stats and feed that to the automated playback tests.

Community

  • [DONE] [marcia] Build community effort to support the Buddy Up project
  • [ON TRACK] [ashughes] Help create at least three small, focused test-day events that focus on helping contributors be more effective
    • we have three events planned this quarter: Bangladesh, Argentina/Spain, and the UK