Releases/Firefox 36/Test Plan
Contents
- 1 Summary
- 2 Features
- 3 Iterations
- 4 Stability
- 5 Bug Triage
- 6 Milestones
- 6.1 Aurora Migration
- 6.2 Betas
Summary
This page is to track testing of Firefox 36 from mozilla-central (Nightly) through to mozilla-release.
Bug queries and graphs for this release and Firefox iterations: Firefox QA Dashboard
How you can help
Would you like to help test Firefox 36? Join us! Here's some things you can do:
- Do some basic setup tasks to get all the tools you need in place.
- Come to the next Bugzilla Tour.
- Set up multiple profiles in Firefox.
- Download and install Nightly (Firefox 36)
- Try some tasks in One and Done: Triage an unconfirmed bug; Verify a fixed bug
- Join our mailing list on dev-quality
- Hang out in our irc channels: irc.mozilla.org #qa and #testday
- If you have questions you can contact Robert Kaiser (:KaiRo)
Schedule
Please consult the Rapid Release Calendar for more information.
Milestone | Date | Checks |
Nightly | 2014-10-13 | Conduct daily stability and bug triage, ensure features are owned |
Aurora Migration | 2014-11-28 | Conduct testing to ensure Firefox 36 Aurora builds are okay, sign off for updates on Friday |
Aurora Updates enabled | 2014-10-17 | Continue daily stability and bug triage, ensure features are tested |
Beta Uplift | 2015-01-12 | Review testing and sign off every beta release |
Release | 2015-02-24 | Review tests and sign off release |
Meetings
Meeting | Purpose | When | Vidyo Room | IRC Backchannel |
Channel Status | Raise quality concerns with the Release Management team | 10:00 Pacific on Tuesday & Thursday | ReleaseCoordination | #planning |
Firefox QA | Meet the Desktop QA team and hear what they're working on | 8:30am Pacific on Wednesdays | DesktopQA | #qa |
QA Team | Meet the entire QA team and hear what they're working on! | 1:30am Pacific on Wednesdays | QA | #qa |
Checks
Priority | Check | Description | Frequency | Last Checked |
1 | Stability | Review the stability reports to identify new/rising crash signatures | Daily | 2014-xx-xx |
2 | Pushlog | Review the pushlog to identify fixed bugs needing verification | Daily | 2014-xx-XX (not keeping up tho) |
2.5 | Iteration | Triage the iteration bugs, add QA contacts | Twice weekly | 2014-xx-XX |
3 | QA Needed Bugs | Review and assign qawanted bugs for investigation | At least twice a week | 2014-xx-XX |
4 | Unconfirmed Bugs | Triage and test unconfirmed bug reports | At least once a week | 2014-xx-XX |
5 | Fixed Bugs | Flag fixed bugs for verification | At least once a week | 2014-xx-XX |
6 | Verification Needed Bugs | Test verifyme bugs to confirm fixed bugs are fixed | At least once a week | 2014-xx-XX |
7 | Reported Bugs | Document and escalated reported bugs as necessary | Every Monday | 2014-xx-XX |
8 | Features | Ensure features are owned and on track for release | Once a week | 2014-xx-XX |
9 | Automation | Ensure automation bugs are owned and on track for resolution | Once a week | 2014-09-XX |
Features
The following tracks features targeted for Firefox 36. For more information consult this wiki.
- Ensure all features which do not require testing are flagged qe-verify-
- Ensure all features which do require testing are flagged qe-verify+ and have an assigned QA Contact.
- Ensure all features are either signed-off or disabled before release
XXX
- Feature owner: Some One
- Status: Unknown
Bugzilla query error
error, http-bad-status, Array
Iterations
Firefox development now works on two-week iterations. This section tracks the bugs selected for development in these iterations. Please consult the Iteration Dev Walkthrough and the Firefox/IterativeDevelopment page for more information on this process.
- Ensure all bugs have qe-verify status indicated.
- Ensure all qe-verify+ bugs have a QA Contact (The Feature Owner page can help here)
- Ensure all bugs important to be verified are VERIFIED before the end of the next iteration
- See the Firefox QA dashboard (link at the start of this page) for bug queries and status.
Stability
Reports
- Daily explosiveness report: Firefox 36
- 7-day top crash report: 36.0a1
- Consult this walkthrough for more information
Bug Triage
Pushlog
The following tracks triage of the Mozilla pushlogs.
- Review each bug in the pushlog and determine if the bug should be
- A) To be tested by the QA community --> add the qe-verify+ flag to the bug and set the status-firefox36 flag to fixed
- B) To be tested by a new volunteer --> add the [good first verify] whiteboard tag
- C) To be tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
- D) Not to be tested at all --> add the qe-verify-flag
- If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing
- Be sure to set the status-firefox36:fixed flag
- Ignore any bugs for non-Desktop platforms, with in-testsuite coverage, or which address test failures
QA Needed Bugs
The following tracks bugs which have requests for urgent QA assistance.
- Choose a bug to test from the list below
- Read it and make sure you understand what information is needed (this may be steps or URLs to reproduce, a regression range, or something else altogether)
- Use the latest build of this Firefox version and test until you've discovered the information necessary
- Report this information to the bug and remove the qawanted keyword
- Document the bug in the Completed list
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Unconfirmed Bugs
The following tracks triage and testing of incoming, unconfirmed bug reports.
- Choose a bug to test from the list below
- Use the latest Nightly build to confirm the bug reproduces
- Flag with reporter with needinfo if you cannot reproduce and need more information
- Set the status to NEW and ensure the bug is in the appropriate component if you can reproduce the bug (provide as much detail about your testing as possible)
- If it's a Firefox bug you may want to add the firefox-backlog? flag to it.
All Unconfirmed Bugs
65 Total; 65 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Unchanged Since One Week
65 Total; 65 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Unchanged Since One Month
65 Total; 65 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Orphaned Fixes
Recent fixes with no target milestone set
- See "Ongoing work" section of the Firefox QA dashboard for a Bugzilla query.
Recent fixes with status-firefox36 missing.
Verification Needed Bugs
- Review the bugs below in the Pending Triage section and select a bug you want to test
- Make sure you understand the bug before you begin testing it
- Test on a previously known broken build to make sure you can reproduce the bug
- Test on the latest build from the branches which are fixed to confirm the bug no longer reproduces
- Mark the bug VERIFIED FIXED if the bug appears to be fixed and set the appropriate status-firefox flag to verified
- Reopen the bug if it still reproduces
- See the Firefox QA dashboard for bug queries and numbers.
Fixed Bugs
The following tracks triage and testing of bugs fixed for this Firefox version.
- Select a bug and review it to determine if it should be
- A) To be tested by the QA community --> add the qe-verify+ flag to the bug and set the status-firefox35 flag to fixed
- B) To be tested by a new volunteer --> add the [good first verify] whiteboard tag
- C) To be tested by the bug reporter --> set the needinfo flag to ask the reporter to confirm the bug is fixed
- D) Not to be tested at all --> add the qe-verify- flag
- If you are unsure, set the needinfo flag to the Assignee or QA Contact and ask if the bug needs testing. If it looks like it may be or should be in the testsuite, add the testsuite? flag.
- Be sure to set the status-firefox35:fixed flag
- See the Firefox QA dashboard for bug queries and numbers.
Reported Bugs
The following tracks bugs reported through day-to-day testing.
- Once per week, Softvision will send an email identifying their found issues
- Make sure the issues are documented below
- Review the NEW issues to make sure they are actionable
- Actionable bugs can be flagged "firefox-backlog?" according to the backlog triage criteria
- Any regressions should be nominated to track and be escalated to Release Engineering
- Review the FIXED issues to make sure they are verified
(To be updated ...)
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Milestones
Aurora Migration
Post-merge Work (as soon as updates are disabled):
- [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the auroratest channel (before first Aurora builds are generated)
Pre-sign-off Checks:
- [DONE] Automation - document all Mozmill test failures (on functional tests) and memory usage regressions (on endurance tests - note that those are only run on nightly, so you look for regressions that happened there). Bugs for those are usually filed by the automation team, but check that they are on top of that.
- [DONE] Features - all scoped features are signed off as ready for Aurora by owners. (If you haven't heard anything to the contrary, that is probably true, but make sure we don't know of blocking issues.)
- [DONE] Bugs - document and review any reported bugs and issues, nominate them for tracking if serious (i.e. make sure all bugs needed to be tracked have at least a request for that up).
- [DONE] Stability - review crash stats before sign-off to raise any red flags.
- [DONE] L10n - contact L10n drivers (usually :Pike) to see if there are any known localization issues.
Post-sign-off Checks (as soon as updates are enabled again):
- [DONE] MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel
- [DONE] Updates - previous versions of Aurora update to the new version (check a few old builds manually once next round of new builds is available)
35-specific steps around Dev Edition landing:
- [DONE] 11/07 - MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the auroratest channel (before first Aurora builds are generated)
- [DONE] 11/10 - MM-CI - configure the mozilla-aurora_update testrun on MM-CI to use the aurora channel
Automation
- Testrun Reports
Report | Test Failures |
Add-ons | |
Functional | |
Remote | |
L10n | |
Updates | |
Endurance |
- Memory Usage Report
Platform | Min Memory Usage | Max Memory Usage | Avg Memory Usage | Net Compared to Firefox 32 |
Windows 32-bit | ||||
Mac OS X | ||||
Linux 32-bit | ||||
Linux 64-bit |
Memory regression being tracked/investigated in: <bug number>
Features
Feature | Owner | Aurora Sign-off Status | Potential Blockers |
name | someone | ||
name |
Updates
Platform | Source Build | Target Build | Result |
Windows | |||
Mac OS X | |||
Linux | |||
Reported Bugs
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Betas
Sign-off emails
- For Beta 1 and RC, sign off on beta-cdntest, RelMan needs to give the "go" for pushing out
- For Betas in between, directly send "go" for pushing when beta-cdntest is OK.
Known Mozmill Failures
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Beta 1
Build Information
- build1: 6231299d9787 (ID: 20150112201205)
- build2: 521859f9eae2 (ID: 20150114125146)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Exploratory testing covering: Web compatibility, Flash, No DevEdition carried from Aurora, In-content Preferences
- [DONE] Bug verification
- [DONE] Exploratory testing covering: Web compatibility, Flash
- [DONE] Verification of bug 1119535 (build2)
- Automated (configs for ondemand, build2)
Bugs Reported
9 Total; 9 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 2
Build Information
- build1: 1b21115851ef (ID: 20150120155007)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Verification of bug 1119535 on Windows XP
- [DONE] Bug verification
Bugs Reported
12 Total; 12 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 3
Build Information
- build1: 5807991ef30a (ID: 20150122214638)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Exploratory testing covering: Audio, Video, Flash
- [DONE] Verification of bug 1119535 on Windows XP
- [DONE] Bug verification
Bugs Reported
12 Total; 12 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 4
Build Information
- build1: 59aa16cfd49f (ID: 20150126151838)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Verification of bug 1119535 on Windows XP
- [DONE] Bug verification
Bugs Reported
4 Total; 4 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 5
Build Information
- build1: ef10110ab600 (ID: 20150129200438)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Exploratory testing for Flash content
- [DONE] Verification of bug 1119535 on Windows XP
- [DONE] Bug verification
Bugs Reported
13 Total; 13 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 6
Build Information
- build1: 47e26f891d66 (ID: 20150202183609)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Exploratory testing for MP4 & Youtube playback
- [DONE] Verification of bug 1119535 on Windows XP
- [DONE] Bug verification
Bugs Reported
ID | Summary | Resolution |
---|---|---|
777680 | "Bookmark This Page" in context menu should be highlighted | FIXED |
1127884 | The "Unresponsive script" warning appears on Facebook page | WORKSFORME |
1128971 | crash in mozilla::image::ProgressTracker::SyncNotify(imgRequestProxy*) | WORKSFORME |
3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 7
Build Information
- build1: 2807adaccd5f (ID: 20150205114429)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Exploratory testing for Youtube playback
- [DONE] Verification of bug 1119535 on Windows XP
- [DONE] Bug verification
After decision to ship only to Mac and Linux, and leave Windows with b6:
Bugs Reported
8 Total; 8 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 8
Build Information
- build1: 1584db7257a6 (ID: 20150209164123)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Verification of bug 1119535 on Windows XP
- [DONE] Bug verification
Bugs Reported
3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 9
Build Information
- build1: 57f79302248f (ID: 20150212154903)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Regression testing
- [DONE] Exploratory testing for Youtube and other Flash content
- [DONE] Bug verification
Bugs Reported
13 Total; 13 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Beta 10
Build Information
- build1: 9e2391999085 (ID: 20150217132925)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Exploratory testing covering: Web Compatibility, Video & Audio playback, Youtube Flash Compatibility
- [DONE] Investigation of bug 1134052 on multiple Windows Vista x86 machines
- [DONE] Verification of bug 1119535 on Windows XP (since Youtube serves HTML5 again for Windows)
- [DONE] Bug verification
Bugs Reported
9 Total; 9 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Release Candidate
Build Information
- build1: 59eefd8af5d7 (ID: 20150219174507)
- build2: 88c5342693e3 (ID: 20150222232811)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Exploratory testing covering: Web Compatibility, Youtube (Flash & HTML5), Flash content
- [DONE] Bug verification
- Automated (configs for ondemand)
- This is the 36.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
- [DONE] Automation Functional tests (report)
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
Build 2 (done because of the AMD crash being >1% in build1 on beta):
- Manual
- [DONE] Sanity testing covering: Basic Web Compatibility, investigation of AMD crash
- Automated (configs for ondemand)
- This is the 36.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
- [DONE] Updates on beta-cdntest channel (report)
- [DONE] Updates on beta channel (report)
- [DONE] Updates on release-cdntest channel (report)
- [DONE] Updates on release channel (report)
Bugs Reported
10 Total; 10 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Release 36.0
Build Information
- See Release Candidate Build 2
Testing
- Update Testing - see Release Candidate Build 2
- Throttle Testing (see Walkthrough for details)
- Used commands:
./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/35.0.1/20150122214805/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
- Results
- After release push (expected to be @ 25%), 2015-02-24: Win32: 25% - Mac: 24% - Linux32: 21% - Linux64: 23%
- After disabling updates (expected to be @ 0%), 2015-02-27: Win32: 0% - Mac: 0% - Linux32: 0% - Linux64: 0%
- After fully enabling updates (expected to be @ 100%), 2015-02-xx: Win32: % - Mac: % - Linux32: % - Linux64: %
- Results
36.0.1
Build Information
- build1: 611f37fccce0 (ID: 20150304131950)
- build2: 1bf1bdefeae0 (ID: 20150305021524)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual - Firefox 36.0.1
- [DONE] Exploratory testing covering: Web Compatibility, Printing, WebGL, Hello
- [DONE] Bug verification
Bugs Reported
8 Total; 8 Open (100%); 0 Resolved (0%); 0 Verified (0%);
36.0.3
Build Information
- build1: f5cca5d1d661 (ID: 20150319201009)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- X beta blocking regression, stability, or support issues discovered
Testing
- Manual - Firefox 36.0.3
- [DONE] Exploratory testing covering: Web Compatibility, Printing, WebGL, Hello
- [DONE] Bug verification
Bugs Reported
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
36.0.4
Build Information
- build1: df45b1c67169 (ID: 20150320202338)
Sign-off Criteria
- Mozmill functional automation ran with/without unexpected failures
- Mozmill update automation ran with/without unexpected failures
- No blocking regression, stability, or support issues discovered
Testing
- Manual
- [DONE] Verify security fixes
Bugs Reported
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);