Releases/Firefox 70/Test Plan

From MozillaWiki
Jump to: navigation, search

Summary

The purpose of this page is to track the testing of Firefox 69 from Nightly to Release, and post Release if needed.

Release schedule

Status meetings

The Channel Status meeting is used to raise quality concerns with the Release Management team.

QA points of contact

Communication channels

Firefox for Desktop

Engineering QA
  • Reporting to: Tom Grabowski, Tania Maity
  • Softvision Manager: Rares Bologa
  • Team Leads: Brindusa Tot
  • Offices: Romania (EEST), Argentina (GMT-3)
  • Responsible for: new feature testing, Nightly build readiness, Nightly build regression testing, bug triage, bug verification, bug investigation
Release QA
  • Reporting to: Tom Grabowski, Tania Maity
  • Softvision Manager: Andrei Vaida
  • Team Leads: Bogdan Maris, Cornel Ionce
  • Office: Romania (EEST)
  • Responsible for: new feature testing, Beta build readiness, Developer Edition build readiness, ESR build readiness, RC build readiness, Dot release build readiness, Chemspill build readiness, bug verification, bug investigation

Firefox for Mobile

Release QA
Firefox for iOS QA

AMO, WebExtensions

Add-ons QA
  • Reporting to: Krupa Raj
  • Softvision Manager: Rares Bologa
  • Team Leads: Victor Carciu, Madalin Cotetiu
  • Office: Romania (EEST)
  • Responsible for: AMO feature testing, AMO weekly release, AMO Sanity, AMO bug verification, Webextensions features, Webextensions bug triage and verification, Telemetry

Experiments, Normandy, Subscription services, and other Firefox features

Experiments QA
  • Reporting to: Krupa Raj
  • Softvision Manager: Rares Bologa
  • Team Leads: Paul Oiegas
  • Office: Romania (EEST)
  • Responsible for: testing Experiments, Normandy, Experimenter website, Activity Stream, Screenshots, Send (Web & Android) and Lockbox Desktop.

QA activities

New feature readiness

All new features tested by QA and shipping with Firefox 69 are available here.

The feature testing process followed by QA is available here.

Bug fix verification

  • Daily verification of uplifted bugs - these are uplifts, triaged daily and verified under ~48hrs
  • Ongoing verification of Nightly bugs - this is the general bug verification done throughout the Nightly cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of qe-verify? bugs
  • Ongoing verification of Beta bugs - this is the general bug verification done throughout the Beta cycle; it sometimes overlaps with uplifts verification; also involves active monitoring of qe-verify? bugs
  • On-demand verification of bugs - these are special requests (ideally coming in through PI, but also through Bugzilla needinfo?, chat or emails)
  • Periodical verification of security bugs - these are post-critsmash bugs, triaged and verified ~every 2wks

Unconfirmed bug triage

This is the triage conducted by Engineering QA.

Security bug triage

This is the post-CritSmash triage conducted by Engineering QA (on Nightly) and Release QA (on Beta, Release, ESR).

Bug investigation

This is the daily check and investigation of new regressionwindow-wanted bugs, conducted by Engineering QA and Release QA.

Nightly 71 milestones

[early/mid cycle] Nightly build readiness

Scope: assess build readiness for shipping to Beta by testing Nightly periodically and providing a list of issues (if any) that need to be addressed before the merge (more on this).

Nightly 69.0a1 (2019-05-30)
  • Build: 69.0a1 (commit, ID) - GREEN
    • [DONE] Functional testing (manual) → report
      • New bugs
        • Regressions: none
        • Non-regressions: 1554912, 1555113
      • Known bugs
        • High-impact: none
        • Low/medium-impact: bug list
Nightly 69.0a1 (2019-06-13)
  • Build: 69.0a1 (commit, ID) - GREEN
    • [DONE] Functional testing (manual) → report
      • New bugs
        • Regressions: none
        • Non-regressions: 1558762
      • Known bugs
        • High-impact: none
        • Low/medium-impact: bug list

[prior to merge] Nightly regression testing

Nightly 69.0a1 (date)
  • Build: 69.0a1 (commit, ID) - Sign off_Status
    • [NOT STARTED] Regression testing (manual) → [report]
      • New bugs
        • Regressions: ID1, ID2, IDn
        • Non-regressions: ID1, ID2, IDn
      • Known bugs
        • High-impact: ID1, ID2, IDn
        • Low/medium-impact: [bug list]

[optional] [prior to merge] Nightly new feature smoke testing

Note: this is optional, based on available QA bandwidth.

Beta/DevEd 70 milestones

Developer Edition build readiness

Scope: assess build stability periodically (around every 2 weeks) to make sure we are shipping quality products to our Developer Edition users from the aurora update channel (more on this).

DevEd 70.0b1
  • Build: 70.0b1-build2 (2c2cb0df9517, 20190827005903) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [BLOCKED] aurora-cdntest → [report]
      • [NOT STARTED] aurora → [report]
DevEd 70.0b2
  • Build: 70.0b2-build2 (04d54c74a744, 20190830040415) - GREEN
    • Functional tests (manual):
    • Update tests (manual):
      • [DONE] aurora-cdntest → report
      • [NOT STARTED] aurora → report
DevEd 70.0b5
  • Build: 70.0b5-build1 (1021b3e283c6, 20190909162732) - GREEN
    • Functional tests (manual):
      • [DONE] Developer Edition Recurring Sign off → TestRail
      • [DONE] [Bug fix verification]
    • Update tests (manual):
      • [DONE] aurora-cdntest (as part of Beta 70.0b5 update tests)
      • [DONE] aurora (as part of Beta 70.0b5 update tests)
DevEd 70.0b9
  • Build: 70.0b9-build1 (48e74f990086, 20190923154733) - GREEN
    • Functional tests (manual):
      • [DONE] Developer Edition Recurring Sign off → TestRail
      • [DONE] [Bug fix verification]
    • Update tests (manual):
      • [DONE] aurora-cdntest (as part of Beta 70.0b9 update tests)
      • [DONE] aurora (as part of Beta 70.0b9 update tests)

Beta build readiness

Beta 70.0b3
  • Build: 70.0b3-build3 (82e9cab54ba65854416b316306a31f045bdb2321, 20190902191027) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Plug-ins, Address Bar and Crash Reporting → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Smoke testing on DLL interceptor → Test Results
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b4
  • Build: 70.0b4-build3 (5da3c0a0771e9c2ff102bfc5d97d9bb5bb93788e, 20190905174512) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Images and Audio/Video → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b5
  • Build: 70.0b5-build1 (1021b3e283c6eb34514b46ad600c42a647ad85a8, 20190909162732) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on WebExtensions, Scrolling, Panning and Zooming and Geolocation → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b6
  • Build: 70.0b6-build1 (bb9376d19d79, 20190912160217) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Safe Browsing and Password Manager & Form Autofill → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b7
  • Build: 70.0b7-build1 (2771f6fe4899, 20190916074538) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Address Bar and Audio/Video → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b8
  • Build: 70.0b8-build1 (45c0e8a9df93f545bfacf07e5a78bd69559c6adf, 20190919164641) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Downloads, Installer and Theme and Toolbar Customization → [1]
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b9
  • Build: 70.0b9-build1 (48e74f990086920697ec7b6d3a6ff95554de390f, 20190923154733) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Password Manager and Search → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b10
  • Build: 70.0b10-build1 (6f62baa8eaaa, 20190926005616) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Startup and Profile and Anti-Tracking → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [IN PROGRESS] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → [ report]
Beta 70.0b11
  • Build: 70.0b11-build1 (e8e31000eab2, 20190930132843) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke and Regression testing on Graphics and Bookmarks → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → [ report]
Beta 70.0b12
  • Build: 70.0b12-build1 (24de1dfb6086, 20191002212950) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke and Regression testing on OS Integration, Data migration and History → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b13
  • Build: 70.0b13-build3 ([2], 20191007220302) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke and Regression testing on PDF Viewer and Printing → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report
Beta 70.0b14
  • Build: 70.0b14-build1 (5d7ecb0646db, 20191010142853) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke and Regression testing on Graphics and Session Restore → TestRail
      • [DONE] Smoke testing on Ubuntu snap package → TestRail
      • [DONE] Bug fix verification – Bug list
    • Update tests (manual):
      • [DONE] beta-cdntest, aurora-cdntest → report
      • [NOT STARTED] beta, aurora → report

Release 70 milestones

RC1 70.0
  • Build: 70.0-build1 (0eae18af659f, 20191014163058) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web Compatibility → Status Report
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [DONE] Bug fix verification – Bug Verification
    • Update tests (manual):
      • [DONE] beta-cdntest → report
      • [NOT STARTED] beta → report
      • [DONE] release-localtest → report
      • [NOT STARTED] release-cdntest → report
      • [NOT STARTED] release → report
RC2 70.0
  • Build: 70.0-build2 (990d9f90f0f5, 20191016161957) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web Compatibility → Status Report
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [DONE] Bug fix verification – Bug Verification
    • Update tests (manual):
      • [DONE] beta-cdntest → report
      • [NOT STARTED] beta → report
      • [DONE] release-localtest → report
      • [DONE] release-cdntest → report
      • [DONE] release → report
RC 70.0.1
  • Build: 70.0.1-build1 (21ce5fdd3146, 20191030021342) - GREEN
    • Functional tests (manual):
      • [DONE] Smoke, Regression and Exploratory testing on Web Compatibility → Status Report
      • [DONE] Smoke testing on Ubuntu snap package → report
      • [DONE] Smoke testing on Windows Acer builds → report
      • [DONE] Bug fix verification – Bug Verification
    • Update tests (manual):
      • [DONE] release-cdntest → report
      • [NOT STARTED] release → report
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_70/Test_Plan&oldid=1219719"