QA/Fennec/9/Nightly
Contents
Fennec 9.0 Nightly Test Plan
Overview
The goal of this Overall Test Plan is to lay out our general testing strategy, determine who will own certain test areas and when they will do it and, finally, track the status. This does not cover individual feature quality work as those will be listed within the features themselves (a list of those features can be found below or on the mobile team's Feature Planning page). All of our releases are tested off of the "Fennec 9.0 Nightly Catch-All Test Run" on the Litmus TCM.
Project contacts
- Dev Lead: Mark Finkle (:mfinkle)
- Build: John Oduinn (joduinn)
- QA: Aaron Train (AaronMT)
Schedule
- Start Date: 08/16/2011
- End Date: 09/27/2011
QA Queries
Build Location and IDs
- Go to the nightly builds folder in our ftp directory
- Go to the "latest-mozilla-central" folder
- Download and install the candidate onto your Android phone/device
Test Areas
Smoketests
QA Task | Person/Group | How Often? |
Smoketests | Waverley | Daily |
Lite verification tests (or smoketests) will be first run when the builds are ready. Tests are available on litmus under the Fennec 9.0 Test Suite. In any case of failure in any Smoketest, blockers need to be raised ASAP before proceeding to the BFTs.
Results details can be found in Waverley's results wiki.
BFTs for 9.0 Features
QA Task | Person/Group | How Often? |
BFTs | Waverley | Daily |
BFTs are a more comprehensive set of litmus tests that cover functional testing of features that have landed on Fennec 7.0. Tests are available on litmus under the Fennec 9.0 Test Suite.
Results details can be found in the Waverley results wiki
OS Integration
QA Task | Person/Group | How Often? |
OS Integration | Waverley | Weekly |
The OS Integration testsuite is a set of tests that encompass device-level and operating-system-level tests for the Android platform. In terms of criteria, they will determine a quality criteria for pass or failure of the system level characteristics of the application.
Results can be found in the Waverley results wiki.
Fixed Bugs Verifications
QA Task | Person/Group | How Often? |
Bug Verification | Waverley | Daily |
The following link contains a list of bugs that are fixed for 9.0 and need verification
Crasher Triage
QA Task | Person/Group | How Often? |
Crasher Triage | mw22 | Daily |
Socorro is Mozilla's primary web interface for triaging crash reports sent in by our user base. Due to the relatively large amount of changes coming in on a nightly basis, there is a need to check for crash reports coming in for every build. The owner of this task will triage through top crashers on a daily basis, file bugs if necessary and find steps to reproduce for those crashers if possible.
Bug Triage
QA Task | Person/Group | How Often? |
Bug Triage | MoCo Team | Weekly |
On average, the "Fennec" product in Bugzilla receives 30-40 bugs filed every week. The QA-Mobile team will be triaging through those bugs on a weekly basis on Mondays at 9am Pacific. The bug query used is for non-triaged bugs filed over the past week.
Testday
QA Task | Person/Group | How Often? |
Testday | QA Mobile | Once |
To continue determining better quality metrics, we'll want to work with our community to find new bugs and performance/stability metrics through the use of a Testday once per cycle per channel (i.e. nightly and beta). The QA owner for this task will send announcements, create a testplan, moderate the channel and post results for the Testday. To see when the next testday is schedule, please take a look at the Test_Days wiki.