QA/Firefox Updates
Contents
Summary
The following is a tracking page for QA coverage of Firefox Updates. Any and all questions should be directed to the QA Lead on this project, Anthony Hughes (irc: ashughes).
Context
With the move to rapid releases, QA made a risk assessment to slip on the 6 week schedule or to rely heavily on automation as a testing strategy. With that, we quickly moved from 100% manual testing to 100% automated testing. The purpose of this document is to build out and communicate a testing strategy which is inclusive of many types of testing without sacrificing time-to-market. Thereby ensuring our users the most secure and painless update experience.
- Current Testing Strategy
- Automation test partial, complete and fallback update paths
- Parsing automated results for any red-flags, but no focused manual spotchecking of updates
- Random slice of 4 previous versions and 4 locales across 9 platforms
- Time to sign-off updates ~1 hour per channel (4 hours from betatest to release)
- Revised Strategy Going Forward
- Ownership: Update component and new Update features to increase visibility into issues so we can react more aggressively
- Automation: test partial, complete and fallback update paths; parse automated results for any red-flags
- Coverage: 4 previous versions, 4 locales, 9 platforms, releasetest/beta/release
- Time to Sign-off: ~1 hour per channel
- Manual: run Smoketest suite and exploratory testing (billboard, what's new pages, places/profile/add-ons/session retention)
- Coverage: 2 previous versions, 2 locales, 4 platforms, betatest
- Time to Sign-off: ~4 hours
- This strategy should not significantly impact time to market as manual testing can happen in advance of and parallel to automation
- Total time to sign-off: ~6 hours from betatest to release
- Long-term Vision
- Running automation across ALL locales
- Expand automation to capture potential UX regressions
- Evolve manual testing in anticipation of and reaction to issues
- Increase QA visibility and aggressiveness to foster a proactive strategy
Firefox Update Mechanism
The following will be used to track how we test Firefox updates in general.
- Litmus Tests
- Automated Tests
- Bugzilla Triage
- etc...
Summary | Ensure a certain level of quality with the update experience in general |
QA Lead | Anthony Hughes (Desktop), Kevin Brosnan (Mobile) |
Manager | Robert Strong |
QA Status | Currently cataloging existing test coverage |
Silent Updates
Summary | Make the update process silent so that there is the minimal user interaction required. |
Status | Most features are in the process of landing or are in development (2011-11-22) |
QA Lead | Anthony Hughes |
Manager | Chris Lee |
QA Status | QA is drafting test plans and test cases; testing landed features (2011-11-22) |
Meetings | Every Thursday at 1pm PDT |
This is an aggregate of several smaller features...(see below)
Feature | Owner | Test Plan | Status | Milestone |
Bypass UAC Dialog on Windows | Simona Badau | test plan | Test Planning | Firefox 12 |
Background Updates | Vlad Ghetiu | test plan | Test Planning | Firefox 13 |
Workflow with Incompatible Add-ons on Update | Ioana Budnar | test plan | n/a | Unscoped |
Add-ons Compatible by Default | Virgil Dicu | test plan | SIGNED-OFF 2012-01-25 | Firefox 10 |
What's New Communication | Mihaela Velimiroviciu | test plan | SIGNED-OFF 2011-11-05 | Firefox 8 |
Reduce Displayed UI | Paul Silaghi | test plan | SIGNED-OFF 2011-11-05 | Firefox 8 |
Current overview of compatibility can be found here
Draft test cases can be found here
Litmus test cases can be found here
Etherpad for staging weekly status can be found here
Meeting Notes
Date | Summary | Notes |
2012-01-26 | Silent Update Status |
UAC Service (Simona)
|
2012-01-05 | Silent Update Status |
UAC Service (Simona)
Default Compatible Add-ons (Virgil)
Background Updates (VladG)
|
2011-12-08 | Silent Update Status |
UAC Dialog Suppression (Simona)
Add-ons Compatible by Default (Virgil)
Background Updates (Vlad)
Incompatible Add-ons Workflow (Ioana)
|
2011-12-01 | Silent Update Status | UAC dialog suppression
|
2011-11-24 | Silent Update Status |
|
2011-11-17 | Silent Update Status |
|
2011-11-10 | Silent Update Status |
|
2011-10-21 | Security Review: Background Updates |
|
2011-10-20 | Silent Update Status |
|
2011-10-13 | Silent Update Status |
|
2011-10-13 | Security Review: OS Dialogs |
|
Add-ons
Feature | Owner | Test Plan | Status | Milestone |
Add-ons Confirm on Upgrade | Vlad Maniac | test plan | SIGNED-OFF | Firefox 8 |
Third Party Add-on Warning | Vlad Maniac | test plan | SIGNED-OFF | Firefox 8 |
Mobile
Feature | Owner | Test Plan | Status | Milestone |