Releases/Firefox 27/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 26 | Firefox 27 | Firefox 28 »

Firefox 27 Test Plan

Summary

Milestone Firefox 27
Status Currently in Beta
Lead Tracy Walker (irc: tracy)
2nd Anthony Hughes (irc: ashughes)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 27 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 27. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

  • 2013.09.16: [DONE] Nightly
  • 2013.10.29: [DONE] Aurora
  • 2013.12.05: [DONE] Pre-Beta feature sign-offs
  • 2013.12.12: [DONE] Beta 1
  • 2013.12.17: [DONE] Beta 2
  • skip: Beta 3
  • 2014.01.07: [DONE] Beta 4
  • 2014.01.10: [DONE] Beta 5
  • 2014.01.14: [DONE] Beta 6
  • 2014.01.17: [DONE] Beta 7
  • 2014.01.21: [DONE] Beta 8
  • 2014.01.24: [DONE] Beta 9
  • 2014.01.28: [DONE] Release Candidate
  • 2014.02.04: [DONE] Final
  • 2014.02.13: [DONE] 27.0.1

Features

Sign-off Criteria
  • All user stories covered by manual or automated tests
  • Manual testing complete with no blocking regressions
  • Automated testing complete with no blocking regressions
  • 0 untriaged unconfirmed bugs
  • 0 untriaged unverified fixes
Status
Feature Owner Documentation Tracked Bugs Aurora Merge Sign-off Pre-beta Sign-off
Firefox Metro Juan Becerra Bug Testing [DONE] Disabled
HTTP Backend Cache Mihaela Velimiroviciu, Catalin Varga bugs Disabled NA
New First Run experience Catalin Varga 1 bug [DONE] Disabled
Add a simple tool to display reflows Petruta Rasa MDN doc 1 bug [DONE] [DONE]
SocialAPI: Allow multiple workers Petruta Rasa 1 bug NA [DONE]
Enable history swipe animations by default Juan Becerra 1 bug NA
Interface for predictive actions No need for QA owner - covered automatically 1 bug NA NA
WebGL Shader Editor No need for QA owner - covered automatically 1 bug NA NA
CSS Inheritance changes No need for QA owner - covered automatically 2 bugs NA NA

Beta

Known Issues

Full Query
ID Summary Priority Status
607586 Realplayer Plugin does not work -- RESOLVED
739043 Can't fill fillable PDF forms with PDF Viewer -- RESOLVED
810636 Poor copy & paste behavior with pdf.js P2 VERIFIED
906057 Delay on WebRTC calls -- RESOLVED
922051 Image freezes on caller side after a short period of time -- RESOLVED
922132 Image freezes on callee side after a while -- RESOLVED
928379 QuickTime controls bar is missing -- RESOLVED
936034 Sound is distorted on one side on 1:1 call P5 RESOLVED
943511 Use fallible allocation in nsSegmentedBuffer::AppendNewSegment -- VERIFIED
943885 Investigate when "+N other downloads" should be shown instead of "Show all downloads" -- NEW
945703 Pasting a download link into the download manager sometimes fails. -- RESOLVED
948944 The download indicator is missing from the applications dock -- RESOLVED

12 Total; 1 Open (8.33%); 9 Resolved (75%); 2 Verified (16.67%);


Regression Testing

Beginning with Fx 27, we've completely decoupled the regression testing from each beta. Instead, coverage of regression testing will take place throughout the beta cycle. The goal for Fx27 will be to finish all the regression testing prior to beta8. Breakpad will only be retested for changes landed in breakpad and/or significant changes in Socorro.

Under Test Win 7 Win 8.1 Mac 10.8 Linux
Breakpad [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1
Mac OSX code-signing N/A N/A [DONE] 27.0b1 N/A
Nvidia 3d vision [MISSED] tracy [MISSED] tracy [MISSED] tracy [MISSED] tracy
Downloads [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1
Session Restore [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1 [DONE] 27.0b1
Plug-in compatibility [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4
Mac OSX 32-bit mode N/A N/A [DONE] 27.0b4 N/A
Web compatibility [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2
Software update for root-owned files [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4 [DONE] 27.0b4
Add-on compatibility [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2 [DONE] 27.0b2
Plug-in compatibility [DONE] 27.0b9 [DONE] 27.0b9 [DONE] 27.0b9 [DONE] 27.0b9
What's New page [DONE] 27.0b5 [DONE] 27.0b5 [DONE] 27.0b5 [DONE] 27.0b5
Guidelines
Additional Testing
  • For Fx 27.0b5 the Delicious service was tested on the following environments: Windows 7 x64 , Ubuntu 12.10 x86 and OS X 10.8. The focus of the testing was to assure proper functionality of the browser while Delicious service and sidebar are enabled.

Issues Found

List any new issues found or old issues reopened. These issues will be triaged and escalated by the release leads.

Full Query
ID Summary Priority Status
791195 Session is not restored after Firefox crash -- RESOLVED
856900 WebGL demos and games are slower on Firefox than on Chrome -- RESOLVED
943885 Investigate when "+N other downloads" should be shown instead of "Show all downloads" -- NEW
947079 Passive mixed content indicator (grey triangle) shows up on full HTTPS sites that have no mixed content -- RESOLVED
948907 Unable to scroll when selected text from textarea and outside of it P5 RESOLVED
948935 Duplicate same tool in developer tools P3 RESOLVED
948944 The download indicator is missing from the applications dock -- RESOLVED
958506 ebay.com - Button is not correctly displayed on Ebay Shopping Cart webpage -- RESOLVED
959553 Pdf with anotations doesn't print properly P3 RESOLVED
959596 PDF with annotations is incorrectly printed from Windows P3 RESOLVED
972347 Disk cache entries from PB window also appear in the regular window -- RESOLVED

11 Total; 1 Open (9.09%); 10 Resolved (90.91%); 0 Verified (0%);


27.0.1

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Regression testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on release channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage

Release Candidate / Final

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Regression testing (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Updates on release channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Nvidia 3d vision, Breakpad end-to-end functionality
  • Platforms: Windows 8.1 64bit, Windows XP 32bit, Mac OSX 10.8, Ubuntu 13.10 32bit
Instructions
  • Run these test
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 9

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Regression testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
Instructions
  • Run this test
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 8

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Regression testing (see below)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing:

Bugs Reported

Full Query
ID Summary Status
926297 Test failure "Search engines drop down open state has been changed. Expected: 'true'" in /testSearch/testAddMozSearchProvider.js RESOLVED
935451 Test failure "Geolocation position is: Position acquisition timed out" in /testGeolocation/testShareLocation.js RESOLVED
951244 [ga-IE] Test failure 'TabView is not open.' in /testTabView/testToggleTabView.js and /testTabView/testTabGroupNaming.js RESOLVED
961997 Test failure 'plugin is undefined' in /restartTests/testAddons_pluginDisabledAfterRestart/test2.js RESOLVED
962062 All download entries of the same file are removed from history when one entry is deleted RESOLVED
962145 Test failure "'Recently Closed Tabs' sub menu has one entry - '0' should equal '1'" in testUndoTabFromContextMenu RESOLVED

6 Total; 0 Open (0%); 6 Resolved (100%); 0 Verified (0%);


Beta 7

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Regression testing (see below)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Focus areas: WebRTC
Instructions
  • Use the Moztrap test as a guideline.
  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on different machines.
  • When finding regressions, report a bug and CC Randell Jesup and Maire Reavy (they can help track down if the regressions are real or not).
    • Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment.
Caller Callee Result
Firefox 27.0b7 on Windows 7 Firefox 27.0b7 on Windows 7 PASS
Firefox 23.0 on Windows 7 Firefox 27.0b7 on Windows XP [FAIL] - bug 906057
Chrome Stable on Windows 7 Firefox 27.0b7 on Windows 8 PASS
Firefox 27.0b7 on Windows 7 Firefox 22.0 on Mac OSX 10.8 [FAIL] - bug 922051
Firefox 27.0b7 on Windows 7 Chrome Stable on Ubuntu 12.04 PASS
Firefox 25.0 on Windows XP Firefox 27.0b7 on Windows 7 [FAIL] - bug 906057
Firefox 27.0b7 on Windows XP Firefox 24.0 on Windows XP [FAIL] - bug 906057
Firefox 27.0b7 on Windows XP Chrome Stable on Windows 8.1 PASS
Chrome Stable on Windows XP Firefox 27.0b7 on Mac OSX 10.7 [FAIL] - bug 936034
Firefox 23.0 on Windows XP Firefox 27.0b7 on Ubuntu 13.04 [FAIL] - bug 906057
Chrome Stable on Windows 8.1 Firefox 27.0b7 on Windows 7 PASS
Firefox 27.0b7 on Windows 8 Chrome Stable on Windows XP PASS
Firefox 22.0 on Windows 8 Firefox 27.0b7 on Windows 8.1 PASS
Firefox 27.0b7 on Windows 8.1 Firefox 26.0 on Windows 8.1 PASS
Firefox 26.0 on Windows 8 Firefox 27.0b7 on Mac OSX 10.8 [FAIL] - bug 936034
Chrome Stable on Windows 8.1 Firefox 27.0b7 on Ubuntu 13.04 PASS
Firefox 27.0b7 on Mac OSX 10.9 Firefox 22.0 on Windows 7 '[FAIL] - bug 922132
Chrome Stable on Mac OSX 10.8 Firefox 27.0b7 on Windows XP PASS
Firefox 22.0 on Mac OSX 10.6 Firefox 27.0b7 on Windows 8 PASS
Firefox 27.0b7 on Mac OSX 10.7 Chrome Stable on Mac OSX 10.8 [FAIL] - bug 936034
Firefox 27.0b7 on Mac OSX 10.8 Firefox 26.0 on Ubuntu 13.04 PASS
Firefox 27.0b7 on Ubuntu 12.04 Chrome Stable on Windows 7 PASS
Firefox 22.0 on Ubuntu 12.04 Firefox 27.0b7 on Windows XP [FAIL] - bug 922051
Chrome Stable on Ubuntu 12.04 Firefox 27.0b7 on Windows 8 PASS
Firefox 27.0b7 on Ubuntu 12.04 Firefox 22.0 on Mac OSX 10.7 PASS
Firefox 27.0b7 on Ubuntu 12.04 Chrome Stable on Ubuntu 13.04 PASS
Firefox 27.0b7 on Windows Vista Chrome Stable on Ubuntu 12.04 PASS
Firefox 27.0b7 on Mac OSX 10.9 Firefox 26.0 on Windows Vista PASS
Firefox 27.0b7 on Windows Vista Firefox 22.0 on Windows 7 [FAIL] - bug 922051

Beta 6

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Regression testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)


Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: PDF Viewer, Find toolbar, App Manager
  • Platforms: Windows XP 32bit, Windows 7 64bit, Mac OSX 10.7, Ubuntu 13.10 64bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 5

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Regression testing (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: What's New page, WebGL, Web Audio
  • Platforms: Windows 8.1 64bit, Windows 7 32bit, Mac OSX 10.8.5, Ubuntu 13.10 32bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 4

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Regression testing (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Plugin Compatibility, Update with root owned files, Mac OSX 32-bit Mode
  • Platforms: Windows 8 64bit, Windows 7 64bit, Mac OSX 10.9, Ubuntu 12.04 32bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 2

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)
  • [DONE] Regression testing (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Secure Browsing, Add-on Compatibility, Social API
  • Platforms: Windows 8 64bit, Windows 7 64bit, Mac OSX 10.9, Ubuntu 12.04 32bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
    • Details related to "Browsing Secure Sites" test scenario can be seen in this etherpad
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template

Beta 1

Build Info

Check List

  • [DONE] Automation Functional tests (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Aurora

Branch Migration

Pre Sign-off Tasks

  • [DONE] Firefox branch migrated (Release Management)
  • [DONE] Mozmill Tests branch migrated (Automation Development)
  • [DONE] Updates throttled on aurora channel (Release Engineering)
  • [DONE] Updates verified throttled on aurora channel (QA)

Sign-off Tasks

  • [DONE] Mozmill Daily functional testruns pass with no failures (report)
  • [DONE] Mozmill Daily auroratest update testruns pass with no failures (report)
  • [DONE] Scope features tested and signed off for Aurora uplift
    • minimal feature work for 27

Post Sign-off Tasks

  • [DONE] Aurora sign-off email sent (QA)
  • [DONE] Updates unthrottled on aurora channel (Release Engineering)
  • [DONE] Updates verified unthrottled on aurora channel (QA)

Nightly

WebRTC

Using the Moztrap test as a guideline, verify that Firefox 27.0a1 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:

  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on *different* machines.
  • Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
  • If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
    • Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller Callee Result
Firefox 27.0a1 on Windows 7 Firefox 27.0a1 on Windows 7 PASS
Firefox 22.0 on Windows 7 Firefox 27.0a1 on Windows XP FAIL (1*)
Chrome Stable on Windows 7 Firefox 27.0a1 on Windows 8 PASS
Firefox 27.0a1 on Windows 7 Firefox 22.0 on Mac OSX 10.8 FAIL (3*)
Firefox 27.0a1 on Windows 7 Chrome Stable on Ubuntu 12.04 FAIL (6*)
Firefox 22.0 on Windows XP Firefox 27.0a1 on Windows 7 FAIL (1*)
Firefox 27.0a1 on Windows XP Firefox 22.0 on Windows XP FAIL (3*)
Firefox 27.0a1 on Windows XP Chrome Stable on Windows 8 PASS (4*)
Chrome Stable on Windows XP Firefox 27.0a1 on Mac OSX 10.8 PASS
Firefox 22.0 on Windows XP Firefox 27.0a1 on Ubuntu 13.04 FAIL (1*)
Chrome Stable on Windows 8 Firefox 27.0a1 on Windows 7 PASS
Firefox 27.0a1 on Windows 8 Chrome Stable on Windows XP PASS
Firefox 22.0 on Windows 8 Firefox 27.0a1 on Windows 8 PASS (5*)
Firefox 27.0a1 on Windows 8 Firefox 22.0 on Windows 8 PASS (4*)
Firefox 22.0 on Windows 8 Firefox 27.0a1 on Mac OSX 10.8 PASS (2*)
Chrome Stable on Windows 8 Firefox 27.0a1 on Ubuntu 13.04 PASS
Firefox 27.0a1 on Mac OSX 10.8 Firefox 22.0 on Windows 7 PASS
Chrome Stable on Mac OSX 10.8 Firefox 27.0a1 on Windows XP PASS (2*)
Firefox 22.0 on Mac OSX 10.8 Firefox 27.0a1 on Windows 8 PASS
Firefox 27.0a1 on Mac OSX 10.8 Chrome Stable on Mac OSX 10.8 PASS
Firefox 27.0a1 on Mac OSX 10.8 Firefox 22.0 on Ubuntu 13.04 PASS
Firefox 27.0a1 on Ubuntu 12.04 Chrome Stable on Windows 7 PASS
Firefox 22.0 on Ubuntu 12.04 Firefox 27.0a1 on Windows XP PASS
Chrome Stable on Ubuntu 12.04 Firefox 27.0a1 on Windows 8 FAIL (6*)
Firefox 27.0a1 on Ubuntu 12.04 Firefox 22.0 on Mac OSX 10.8 FAIL (3*)
Firefox 27.0a1 on Ubuntu 12.04 Chrome Stable on Ubuntu 13.04 PASS

Detailed Results:

  • (1*) Image freezes after 2 minutes or less on callee side (the caller video freezes and the audio is working)
    • not reproducible with Firefox 22 and 25.0a2
    • bug 922132
  • (2*) callee hears delayed sound (~5seconds)
    • reproducible with Firefox 22.0
  • (3*) image freezes on caller side
    • not reproducible with Firefox 22.0, 25beta2
    • reproducible with Firefox 26.0a2
    • bug 922051
  • (4*) Caller hears delayed sound
    • reproducible with Firefox 22.0
  • (5*) Callee hears delayed sound
    • reproducible with Firefox 22.0
  • (6*) Caller and callee remain in "Connecting..." state
    • reproducible with Firefox 22 and 26.0a2

Bug Triage

Query Description Remaining
QAWANTED Bugs requiring immediate investigative assistance 1
VERIFYME Bugs with fixes requiring immediate verification
FIXED Bugs with fixes which should be verified as soon as possible
REGRESSION Bugs requiring identification of the first affected build
UNCONFIRMED Bugs requiring confirmation if they are reproducible in a current build
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_27/Test_Plan&oldid=970690"