Releases/Firefox 26/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 25 | Firefox 26 | Firefox 27 »

Firefox 26 Test Plan

Summary

Milestone Firefox 26
Status Currently in Nightly -- merging to Aurora on 2013-09-16
Lead Juan Becerra (irc: juanb)
2nd Anthony Hughes (irc: ashughes)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 26 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 26. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Schedule

  • 2013.08.05: Nightly
  • 2013.09.20: Aurora
  • 2013.10.25: Pre-Beta feature sign-offs
  • 2013.10.31: Beta 1
  • 2013.11.05: Beta 2
  • 2013.11.07: Beta 3
  • 2013.11.12: Beta 4
  • 2013.11.14: Beta 5
  • 2013.11.19: Beta 6
  • 2013.11.21: Beta 7
  • 2013.11.26: Beta 8
  • 2013.11.28: Beta 9
  • 2013.12.03: Beta 10
  • 2013.12.09: Release Candidate
  • 2013.12.10: Final

Features

Sign-off Criteria
  • All user stories covered by manual or automated tests
  • Manual testing complete with no blocking regressions
  • Automated testing complete with no blocking regressions
  • 0 untriaged unconfirmed bugs
  • 0 untriaged unverified fixes
Status
Feature Owner Documentation Tracked Bugs Aurora Merge Sign-off Pre-beta Sign-off
Firefox Metro Juan Becerra Bug Testing [DONE] Disabled
Updates for limited accounts Alexandra Lucinet Test Plan 1 bug [DONE] [DONE]
H264/AAC/MP3 support - MP3 decoding on XP Cornel Ionce Test Plan 1 bug [DONE] [DONE]
Web App Runtime Bogdan Maris Test Plan 11 bugs [DONE] [DONE]
Download Manager (backend work) Mihaela Velimiroviciu, Mihai Morar 16 bugs [DONE] [DONE]
Session Restore (backend work) Cornel Ionce 19 bugs [DONE] [DONE]
JSON Add-ons DB Tracy Walker, Mihai Morar 1 bug [DONE] NA
Make plugins default to click-to-play Paul Silaghi Test Plan 1 bug [DONE] [DONE]
App Manager Petruta Rasa MDN doc NA [ON TRACK]

Nightly

Aurora

Branch Migration

Pre Sign-off Tasks

  • [DONE] Firefox branch migrated (Release Management)
  • [DONE] Mozmill Tests branch migrated (Automation Development)
  • {{|}} Updates throttled on aurora channel (Release Engineering)
  • {{|}} Updates verified throttled on aurora channel (QA)

Sign-off Tasks

  • [DONE] Mozmill Daily functional testruns pass with no failures (report)
  • [DONE] Mozmill Daily auroratest update testruns pass with no failures (report)
  • [DONE] Scope features tested and signed off for Aurora uplift
  • Metro Firefox will be enabled in Aurora for the first time. Two of the issues blocking it have been resolved and verified. See {{bug|903866} and bug 918107 which deal with software keyboard functionality and using a "whatsnew" page that points to a support article.

Post Sign-off Tasks

  • Aurora sign-off email sent (QA)
  • Updates unthrottled on aurora channel (Release Engineering)
  • Updates verified unthrottled on aurora channel (QA)

WebRTC

Using the Moztrap test as a guideline, verify that Firefox 26.0a2 is no worse than Firefox 22.0 when making AppRTC calls of >5 minutes in length. When testing, here are the guidelines to follow:

  • Have only one browser open on each machine at any given time
  • Have only one call running at any given time on a particular machine
  • Make sure the caller and callee are always on *different* machines.
  • Please don't test any 3+-way calls for this sanity check -- We just want to see the results for 1:1 (basic) calling
  • If you find regressions, report a bug and CC Randell Jesup and Maire Reavy; they can help track down if the regressions are real or not
    • Tip: be sure to provide extremely detailed steps to reproduce and witnessed results, as well as detailed information about your test environment; more information is better than not enough information.
Caller Callee Result
Firefox 26.0a2 on Windows 7 Firefox 26.0a2 on Windows 7 PASS
Firefox 22.0 on Windows 7 Firefox 26.0a2 on Windows XP FAIL (6*)
Chrome Stable on Windows 7 Firefox 26.0a2 on Windows 8 PASS
Firefox 26.0a2 on Windows 7 Firefox 22.0 on Mac OSX 10.8 FAIL (2*)
Firefox 26.0a2 on Windows 7 Chrome Stable on Ubuntu 12.04 FAIL (4*)
Firefox 22.0 on Windows XP Firefox 26.0a2 on Windows 7 FAIL (6*)
Firefox 26.0a2 on Windows XP Firefox 22.0 on Windows XP FAIL (2*)
Firefox 26.0a2 on Windows XP Chrome Stable on Windows 8 PASS (5*)
Chrome Stable on Windows XP Firefox 26.0a2 on Mac OSX 10.8 PASS
Firefox 22.0 on Windows XP Firefox 26.0a2 on Ubuntu 13.04 FAIL (6*)
Chrome Stable on Windows 8 Firefox 26.0a2 on Windows 7 PASS
Firefox 26.0a2 on Windows 8 Chrome Stable on Windows XP PASS
Firefox 22.0 on Windows 8 Firefox 26.0a2 on Windows 8 PASS (1*)
Firefox 26.0a2 on Windows 8 Firefox 22.0 on Windows 8 PASS (5*)
Firefox 22.0 on Windows 8 Firefox 26.0a2 on Mac OSX 10.8 PASS(5*)
Chrome Stable on Windows 8 Firefox 26.0a2 on Ubuntu 13.04 PASS
Firefox 26.0a2 on Mac OSX 10.8 Firefox 22.0 on Windows 7 PASS (5*)
Chrome Stable on Mac OSX 10.8 Firefox 26.0a2 on Windows XP PASS
Firefox 22.0 on Mac OSX 10.8 Firefox 26.0a2 on Windows 8 PASS
Firefox 26.0a2 on Mac OSX 10.8 Chrome Stable on Mac OSX 10.8 PASS
Firefox 26.0a2 on Mac OSX 10.8 Firefox 22.0 on Ubuntu 13.04 PASS
Firefox 26.0a2 on Ubuntu 12.04 Chrome Stable on Windows 7 PASS
Firefox 26.0 on Ubuntu 12.04 Firefox 26.0a2 on Windows XP PASS (1*)
Chrome Stable on Ubuntu 12.04 Firefox 26.0a2 on Windows 8 FAIL (3*)
Firefox 26.0a2 on Ubuntu 12.04 Firefox 22.0 on Mac OSX 10.8 FAIL (6*)
Firefox 26.0a2 on Ubuntu 12.04 Chrome Stable on Ubuntu 13.04 PASS

Detailed Results:

  • (1*) Callee sometimes hears the caller with a delay (~1sec)
    • reproducible also with 25.0a2
  • (2*) Image freezes on caller side after ~15 seconds
    • not reproducible with Firefox 22
    • bug 922051 Image freezes on caller side after a short period of time
  • (3*) Caller and callee see only themselves in both video windows (large and small); there is no sound connection between the 2 parts
    • reproducible with Firefox 22
  • (4*) Caller and callee remain in "Connecting..." state
    • reproducible with Firefox 22 and 25.0a2
  • (5*) Caller hears delayed sound
    • reproducible with Firefox 22 and 25beta1
  • (6*) Image freezes on callee side
    • not reproducible with Firefox 22 and 25.0a2
    • bug 922132 Image freezes on callee side after a while

Beta

  • Click to Play by default - check plugin heavy websites for failures and provide feedback on UX

Beta 1

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Breakpad, Click-to-Play Plugins, Session Restore, MP3 playback
  • Platforms: Windows XP, Windows 7 64-bit, Windows Vista, Windows 8.1 32-bit, Mac OSX 10.9, Ubuntu 13.10 64-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • When you get to the WebRTC smoketest refer to the Special Coverage for WebRTC section below for coverage details
  • If you encounter a bug, report it and add it to the list below using the bug number template

If you find a site where click-to-play is broken please add it to this document

Notes

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

  • bug 932786 - CTP doorhanger is not showing the right corresponding pop-up
  • bug 932813 - Time progress jumps to the end for mp3s via network
  • bug 932777 - "Firefox is already running... " error is displayed when trying to open a link .
  • bug 932824 - The overlay to Activate Adobe Flash is briefly displayed
  • bug 932832 - Click to play overlay not showing for Google Talk plugin.

Beta 2

Build Info

Check List

  • [DONE] Automation (report)
    • Investigating a couple of cookie test failures in pl and ru. ... It turned out to be bug 856084
  • [DONE] Regression Testing (report)
    • No new issues reported.
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Session Restore (bug 931891), WebAudio, Downloads.jsm, Updates for limited accounts
  • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.7, Ubuntu 12.04 32-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • If you encounter a bug, report it and add it to the list below using the bug number template
Notes

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

Beta 3

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage

(bug 933156,bug 898291,bug 924286), Mixed Content Blocker, WebApp Runtime

  • Platforms: Windows XP, Windows 7 64-bit, Mac OSX 10.8, Ubuntu 12.04 64-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • If you encounter a bug, report it and add it to the list below using the bug number template
Notes

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

Beta 4

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
    • Mozmill testing infrastructure is experiencing hiccups, but from the partial results, it looks like updates are working fine.
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Graphics (CSS gradients, SVG), layout, TLS, geolocation, PDF Viewer
  • Platforms: Windows XP, Windows 8.1 64bit, Windows 7 32bit, Mac OSX 10.6, Ubuntu 12.04 32-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • If you encounter a bug, report it and add it to the list below using the bug number template
Notes

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

  • bug 937663 - The pdf embedded using object tag is blocked by the mixed content blocker

Beta 5

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: History
  • Platforms: Windows XP, Windows 7 64bit, Mac OSX 10.7, Ubuntu 12.04 64-bit
Instructions
  • Perform exploratory testing on the History component and add the details of your testing here.
  • If you encounter a bug, report it and add it to the list below using the bug number template.
Notes

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

Beta 6

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: SVG, DOM:Core and HTML, Core:Layout, SocialAPI, WebGL
  • Platforms: Windows XP, Windows 7 64bit, Mac OSX 10.8, Ubuntu 13.04 64-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted (including testing more sites for the web compatibility test)
  • If you encounter a bug, report it and add it to the list below using the bug number template
Notes

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

  • bug 940378 - Multiple issues with Social API sidebar

Beta 7

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing
    • Results: A couple of minor issues were found, and an existing SVG performance issue was identified.
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via exploratory testing.

Coverage
Instructions
  • Perform exploratory testing on CTP and SVG starting from the bugs above.
  • Add the details of your testing here.
  • If you encounter a bug, report it and add it to the list below using the bug number template
Notes

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

  • bug 942834 - Loading message is stuck on SVG animation
  • bug 942860 - Notification bar for hidden plugins shows for a few seconds and then hides for www.solomid.net page.

Beta 8

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [ON TRACK] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Installer (bug 941397), Downloads Panel, WebRTC
  • Platforms: Windows 8.1 64bit, Windows 7 32bit, Mac OSX 10.9, Ubuntu 13.10 32-bit & Ubuntu 12.04 32-bit (for WebTRC)
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template
Installer coverage
  • Windows 7 64bit, 8.1 64bit, Windows XP 32bit, Windows Vista 32bit
  • ja, zh-TW, zh-CN, ko, th, he, ru, hi-IN, ar
WebRTC coverage
Caller Callee Notes
Firefox 26.0b8 Windows 8.1 Firefox 26.0b8 Windows 7
Firefox 26.0b8 Windows 8.1 Firefox 26.0b8 Ubuntu 12.04
Firefox 26.0b8 Mac OS X 10.9 Firefox 26.0b8 Windows 8.1
Firefox 26.0b8 Ubuntu 12.04 Firefox 26.0b8 Windows 7
Firefox 26.0b8 Windows 7 Firefox 26.0b8 Mac OS X 10.9
Firefox 26.0b8 Mac OS X 10.9 Firefox 26.0b8 Ubuntu 12.04
Firefox 26.0b8 Windows 7 Chrome stable Windows 8.1
Chrome stable Ubuntu 12.04 Firefox 26.0b8 Windows 8.1
Firefox 26.0b8 Windows 8.1 Chrome stable Mac OS X 10.9
Chrome stable Windows 7 Firefox 26.0b8 Ubuntu 12.04
Chrome stable Mac OS X 10.9 Firefox 26.0b8 Windows 7
Chrome stable Ubuntu 12.04 Firefox 26.0b8 Mac OS X 10.9
Firefox 26.0b8 Windows 8.1 Chrome stable Windows 8.1
Chrome stable Windows 7 Firefox 26.0b8 Windows 7
Firefox 26.0b8 Mac OS X 10.9 Chrome stable Mac OS X 10.8.5
Chrome stable Ubuntu 12.04 Firefox 26.0b8 Ubuntu 12.04

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

  • bug 943634 - stub installer for he needs to be localized
  • bug 943824 - hi-IN stub installer is only partially localized
  • bug 943832 - hi-IN installer isn't localized
  • bug 943836 - Installers and the browser itself display glyphs instead of actual font type for specific languages
  • bug 943383 - 'Allow now' becomes 'allow and remember' if looking again in the CTP doorhanger
  • bug 943885 - Inaccurate number of downloads is displayed on the download panel.
  • bug 943843 - The download size reported by Firefox is much smaller than the actual size on the disk when download size is not known initially.

Beta 9 (skipped)

Beta 10

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [DONE] Updates on releasetest channel (report)
  • [DONE] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template
Notes
  • The sites tested for the Browsing secure sites and Remove trust for extended validation root certificates tests can be found here.

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

  • bug 945703 - Pasting a download link into the download manager sometimes fails.
  • bug 945707 - Pausing a download fails when done after a retry

Release Candidate

Build Info

Check List

  • [DONE] Automation (report)
  • [DONE] Regression Testing (report)
  • [DONE] Updates on betatest channel (report)
  • [ON TRACK] Updates on releasetest channel (report)
  • [ON TRACK] Updates on beta channel (report)

Regression Testing

The following areas have been identified for regression testing and will be covered via Moztrap smoketests.

Coverage
  • Tests: Breakpad, Partner repack update, Windows share compatibility, Mac OSX signing, Browsing secure sites
  • Platforms: Windows 8.1 64-bit, Windows XP 32-bit, Mac OSX 10.9 (32-bit mode), Ubuntu 13.10 32-bit
Instructions
  • Run these tests
  • Select one of the platforms
  • Follow the steps in each of the tests and report your results within Moztrap
  • Make sure to also do some exploratory testing where permitted
  • If you encounter a bug, report it and add it to the list below using the bug number template
Notes
  • The list of secure sites tested is available here.

Issues Found

List any new issues found or old issues reopened using the bug number template. These issues will be triaged and escalated by the release leads.

  • bug 947196 - Cannot play video uploaded in google drive
  • bug 947207 - "Trash" button's label overlaps the menu items on Yahoo Mail

Bug Triage

Query Description Remaining
QAWANTED Bugs requiring immediate investigative assistance 0
VERIFYME Bugs with fixes requiring immediate verification
FIXED Bugs with fixes which should be verified as soon as possible
REGRESSION Bugs requiring identification of the first affected build
UNCONFIRMED Bugs requiring confirmation if they are reproducible in a current build

Known Issues

Bug QA Contact Status
799315 RESOLVED FIXED
745187 NEW
870779 NEW
804580 RESOLVED DUPLICATE
922639 NEW
929067 NEW
931790 NEW
832271 RESOLVED DUPLICATE
493124 NEW
872413 NEW
856900 NEW
900463 NEW
928379 NEW
799663 Cornel Ionce NEW
865208 Cornel Ionce NEW
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_26/Test_Plan&oldid=782283"