Releases/Firefox 74/Test Plan
Contents
Summary
The purpose of this page is to track the testing of Firefox 74 from Nightly to Release, and post Release if needed.
Release schedule
- Release Scheduling calendar
- Rapid Release calendar
Status meetings
The Channel Status meeting is used to raise quality concerns with the Release Management team.
- When: every Tuesday (10:00 AM Pacific) and Thursday (08:00 AM Pacific)
- Where: ReleaseCoordination (Zoom)
- IRC backchannel: #planning
- Meeting notes: notes archive
QA points of contact
Communication channels
- Slack: #qa-coordination
Firefox for Desktop
- Reporting to: Tom Grabowski, Tania Maity
- Softvision Manager: Rares Bologa
- Team Leads: Brindusa Tot
- Offices: Romania (EEST), Argentina (GMT-3)
- Responsible for: new feature testing, Nightly build readiness, Nightly build regression testing, bug triage, bug verification, bug investigation
- Reporting to: Tom Grabowski, Tania Maity
- Softvision Manager: Andrei Vaida
- Team Leads: Bogdan Maris, Cornel Ionce
- Office: Romania (EEST)
- Responsible for: new feature testing, Beta build readiness, Developer Edition build readiness, ESR build readiness, RC build readiness, Dot release build readiness, Chemspill build readiness, bug verification, bug investigation
Firefox for Mobile
- Reporting to: No-Jun Park
- Softvision Manager: Ioana Chiorean
- Team Lead: Sorina Florean
- Office: Romania (EEST)
- Responsible for: todo
- Reporting to: No-Jun Park
- Softvision Manager: Ioana Chiorean
- Team Lead: Catalin Suciu
- Office: Romania (EEST)
- Responsible for: todo
AMO, WebExtensions
- Reporting to: Krupa Raj
- Softvision Manager: Rares Bologa
- Team Leads: Victor Carciu, Madalin Cotetiu
- Office: Romania (EEST)
- Responsible for: AMO feature testing, AMO weekly release, AMO Sanity, AMO bug verification, Webextensions features, Webextensions bug triage and verification, Telemetry
Experiments, Normandy, Subscription services, and other Firefox features
- Reporting to: Krupa Raj
- Softvision Manager: Rares Bologa
- Team Leads: Paul Oiegas
- Office: Romania (EEST)
- Responsible for: testing Experiments, Normandy, Experimenter website, Activity Stream, Screenshots, Send (Web & Android) and Lockbox Desktop.
QA activities
New feature readiness
All new features tested by QA and shipping with Firefox 74 are available here.
The feature testing process followed by QA is available here.
Bug fix verification
- Daily verification of uplifted bugs - these are uplifts, triaged daily and verified under ~48hrs
- Ongoing verification of Nightly bugs - this is the general bug verification done throughout the Nightly cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of
qe-verify?
bugs - Ongoing verification of Beta bugs - this is the general bug verification done throughout the Beta cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of
qe-verify?
bugs - On-demand verification of bugs - these are special requests (ideally coming in through PI, but also through Bugzilla
needinfo?
, chat or emails) - Periodical verification of security bugs - these are post-critsmash bugs, triaged and verified ~every 2wks
Unconfirmed bug triage
This is the triage conducted by Engineering QA.
Security bug triage
This is the post-CritSmash triage conducted by Engineering QA (on Nightly) and Release QA (on Beta, Release, ESR).
Bug investigation
This is the daily check and investigation of new regressionwindow-wanted
bugs, conducted by Engineering QA and Release QA.
Nightly 74 milestones
[early/mid cycle] Nightly build readiness
Scope: assess build readiness for shipping to Beta by testing Nightly periodically and providing a list of issues (if any) that need to be addressed before the merge (more on this).
- Build: 74.0a1 (commit, ID) - N.A.
- [DONE] Functional testing (manual) → [- report]
- New bugs
- Regressions: --
- Non-regressions: --
- Known bugs
- High-impact: --
- Low/medium-impact: [link bug list]
- New bugs
- [DONE] Functional testing (manual) → [- report]
- Build: 73.0a1 (commit, ID) - N.A.
- [NOT STARTED] Functional testing (manual) → report
- New bugs
- Regressions: --
- Non-regressions: --
- Known bugs
- High-impact: --
- Low/medium-impact: [link bug list]
- New bugs
- [NOT STARTED] Functional testing (manual) → report
[prior to merge] Nightly regression testing
- Build: 74.0a1 (commit, ID) - Sign off_Status
- [NOT STARTED] Regression testing (manual) → [report]
- New bugs
- Regressions: ID1, ID2, IDn
- Non-regressions: ID1, ID2, IDn
- Known bugs
- High-impact: ID1, ID2, IDn
- Low/medium-impact: [bug list]
- New bugs
- [NOT STARTED] Regression testing (manual) → [report]
[optional] [prior to merge] Nightly new feature smoke testing
Note: this is optional, based on available QA bandwidth.
Beta 74 milestones
Beta/DevEdition build readiness
Developer Edition build readiness
- Build: 74.0b5-build1 (b07664d2bf28, 20200218224219) - GREEN
- Functional tests (manual):
- [DONE] Smoke, Regression and Exploratory testing on Startup and Profile , Build Branding , WebExtensions , Theme and Toolbar Customization , Web Compatibility, Sync & Firefox Account, OS Integration → TestRail
- Functional tests (manual):
Beta build readiness
- Build: 74.0b1-build2 (2a94604e87b3, 20200210140608) - Done
- Functional tests (manual):
- [DONE] Smoke, Regression and Exploratory testing on Tabbed Browser, OS Integration, User Profile, Form Autofill, Bookmarks → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b2-build1 (2a94604e87b3, 20200210140608) - GREEN
- Functional tests (manual):
- [DONE] Smoke, Regression and Exploratory testing on Startup and Profile, Address Bar and Search, Downloads, Security and Privacy - Cryptomining and Security and Privacy - Anti-Tracking: Anti-Fingerprinting in Standard Mode → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b3-build1 (64dab3403abc, 20200214015126) - GREEN
- Functional tests (manual):
- [DONE] Smoke, Regression and Exploratory testing on Windows Child Mode Functionality, Protection Panel, Protections Report, Protections Report, Zooming, Geolocation and Drag and Drop → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b4-build1 (dcc5fa49cdd3, 20200216164042) - GREEN
- Functional tests (manual):
- [IN PROGRESS] Smoke, Regression and Exploratory testing on Installer, Playback, Block All Autoplay (Fx69+), Global Default Zoom Level (Fx73+), DNS over HTTPS(Provider Selection), and Tabbed Browser → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b5-build1 (b07664d2bf28, 20200218224219) - GREEN
- Functional tests (manual):
- [DONE] Smoke testing on Printing, Theme and Toolbar Customization, PDF Viewer, Bookmarks and History and Password Manager (Autocomplete and Autofill) → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b6-build1 (23159c7ee39f, 20200221001238) - GREEN
- Functional tests (manual):
- [DONE] Smoke testing on Audio/Video, Pocket New Tab, Security and Privacy and Images → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b7-build1 (23159c7ee39f, 20200221211950) - GREEN
- Functional tests (manual):
- [DONE] Smoke testing on Web Compatibility, DNS over HTTPS, Form Autofill, Password Manager 2.0 and Browser Window → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b8-build1 (588c2ea64180, 20200226031638) - GREEN
- Functional tests (manual):
- [DONE] Smoke testing on Uninstaller, Form Autofill with Credit Card Support and Password Manager 2.0 → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
- Build: 74.0b9-build1 (40195b558487, 20200227210932) - GREEN
- Functional tests (manual):
- [DONE] Smoke testing on Preferences, OS Integration and Sync & Firefox Account → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- [DONE] beta, aurora → report
- Functional tests (manual):
RC build readiness
- Build: 74.0-build1 (e98fd13a1a1a, 20200302184608) - Done
- Functional tests (manual):
- [DONE] Smoke, Regression and Exploratory testing on Data Migration, OS Integration, User Profile, Form Autofill, Notifications, Push Notifications and Alerts, Menus, Pre-release telemetry data collection → TestRail
- [DONE] Bug fix verification
- Update tests (manual):
- Functional tests (manual):
- Build: 74.0-build2 (dc1684693eed684bdafeff0ad78d4f6d288fcae1, 20200304223017) - Done
- Functional tests (manual):
- [DONE] Smoke, Regression and Exploratory testing on Telemetry and Web Compatibility → TestRail
- [IN PROGRESS] Bug fix verification
- Update tests (manual):
- Functional tests (manual):
- Build: 74.0b-build3 ([link TBD], ID TBD) - Done
- Functional tests (manual):
- [NOT STARTED] Smoke, Regression and Exploratory testing on Web Compatibility → TestRail
- [NOT STARTED] Bug fix verification
- Update tests (manual):
- Functional tests (manual):