Releases/Firefox 28/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 27 | Firefox 28 | Firefox 29 »

The following documents the testing strategy for Firefox 28 across desktop platforms prior to release on March 18, 2014.

Contents

Contacts

Schedule

  • 2013-10-08: Firefox 28 merged to mozilla-central
  • 2013-11-14: Testday for Australis on the UX Branch
  • 2013-12-09: Firefox 28 merged to mozilla-aurora
  • 2013-12-13: Firefox 28 released to Aurora testers
  • 2014-02-03: Firefox 28 merged to mozilla-beta
  • 2014-02-06: Firefox 28 Beta 1 released
  • 2014-02-11: Firefox 28 Beta 2 released
  • 2014-02-14: Firefox 28 Beta 3 released
  • 2014-02-19: Firefox 28 Beta 4 released
  • 2014-02-25: Firefox 28 Beta 6 released
  • 2014-02-27: Testday for Firefox 28 Beta
  • 2014-02-28: Firefox 28 Beta 7 released
  • 2014-03-04: Firefox 28 Beta 8 released
  • 2014-03-07: Firefox 28 Beta 9 released
  • 2014-03-10: Firefox 28 merged to mozilla-release
  • 2014-03-11: Firefox 28 RC 1 released to beta testers
  • 2014-03-18: Firefox 28 released to general public
  • 2014-03-20: Firefox 28 updates throttled for stabalization
  • 2014-03-28: Firefox 28 updates unthrottled <-- NEXT MILESTONE
  • Release Notes

Features

Disabled

The following features have been deemed not ready for release and will be delayed to Firefox 29 or later.

  • Australis (Cornel Ionce)
  • Disallow Enumeration of navigator.plugins (Catalin Varga)
  • <input type="color"> (Manuela Muntean)
  • Gamepad API (Alexandra Lucinet) -- delayed due to bug 980876
  • WebVTT (Alexandra Lucinet) -- delayed to Firefox 29 (note: this was not on QA's radar until Beta)
  • Metro (Kamil Jozwiak)
  • Multi-tab Closure Improvements (Alexandra Lucinet)
  • Shumway (Matt Wobensmith)

Signed Off

The following features have no blockers and have been deemed ready for release.

Bugs

The following tracks important issues which have potential for negative impact on this Firefox release.

Statistics

NEW

  • 36 fixed
  • 22 closed without fixes
  • 89 remain opened

FIXED

  • 144 verified by target milestone
  • 166 verified by status flag

Regressions

Wont Fix

The following regressions which will not be fixed in this release:

  • bug 977520 Rendering issue with 1/1 Direct3D 11 (OMTC) enabled

Top Crashes

Resolved

The following bugs have been fixed

  • bug 973344 crash in nsTextEditorState::SetValue(nsAString_internal const&, bool, bool)
  • bug 970483 gfxDWriteFont::ComputeMetrics(gfxFont::AntialiasOption)
  • bug 966543 crashes at MacIOSurface::GetDevicePixelWidth() with OMTC
  • bug 963316 crashes in JSObject::is<js::ArgumentsObject>()
  • bug 957258 crash in sprotector.dll
  • bug 952321 crash in mozilla::dom::DocumentBinding::genericMethod

Wont Fix

The following bugs will not be fixed in this release

  • bug 976141 crash in mozalloc_abort(char const* const) @ GCGraphBuilder::NoteChild(void*, nsCycleCollectionParticipant*, nsCString)
  • bug 976125 crash in nsCSSStyleSheet::GetOwningDocument()
  • bug 973138 startup crash in __fnNCDESTROY (mostly with MovieMode.48CA2AEFA22D.dll)
  • bug 967985 crashes with image discarding and decode-on-draw disabled
  • bug 962846 crash in mozalloc_abort(char const*) @ GCGraphBuilder::DescribeRefCountedNode(unsigned int, char const*)
  • bug 960692 Metro Firefox crash in igd10iumd32.dll@0xbc69 within mozilla::layers::CompositorD3D11::BeginFrame
  • bug 951906 Crashes reading native app preferences, mostly on startup, mostly on OS X 10.9 and 10.9.1, with thunder@xunlei.com extension
  • bug 950895 crash in mozalloc_abort(char const* const) @ mozilla::BaseMediaResource::DispatchBytesConsumed(__int64, __int64)
  • bug 943511 Use fallible allocation in nsSegmentedBuffer::AppendNewSegment
  • bug 943021 Safebrowsing service should use fallible arrays for its buffers
  • bug 928168 [10.9] Crashes @ libsystem_kernel.dylib@0x15866
  • bug 897870 crash in mozilla::Logger::~Logger with abort message (Pickle::Pickle)
  • bug 888605 [10.9] crash in [NSPopUpButtonObserver menuChangedItem] with NSUnknownKeyException error message
  • bug 814954 crash in JSC::Yarr::YarrGenerator<int>::opCompileParenthesesSubpattern
  • bug 812186 crash in js::InvokeKernel @ UserCallWinProcCheckWow with Norton Confidential 2013
  • bug 805406 crash in gfxContext::PushClipsToDT with Direct2D 1.1 (d3d11.dll 6.2 or 6.3)
  • bug 798274 crash in gfxContext::PushGroupAndCopyBackground with Direct2D 1.1 (d3d11.dll 6.2)

Unactionable

The following bugs remain top issues but will remain unfixed unless new information or means of investigation becomes known

  • bug 858032 crashes in EnterBaseline
  • bug 787879 crash in js::SweepBackgroundThings
  • bug 761081 crash in JSScript::markChildren
  • bug 719114 Firefox Crash @ js::GCMarker::processMarkStackTop
  • bug 711568 Firefox Crash Reports [@ EMPTY: no crashing thread identified; corrupt dump ]
  • bug 668583 crash @ js::gc::ScanRope

Nightly

The following tracks testing of Firefox 28 on the mozilla-central branch.

Fixed Bug Verifications

  • Download and install the latest Firefox Nightly
  • Load this list of bug reports
  • Read each bug report until you find one you understand and want to test
  • Download and install the version of Firefox used to report the bug
  • Using the older build, test the claim made in the bug report to see if you can reproduce it
    • If you cannot reproduce the bug move on to another bug or ask the reporter to verify the fix
  • Using the latest Nightly, test to see if you can reproduce the bug
    • If you cannot reproduce the bug set the STATUS to VERIFIED FIXED and add a comment describing your testing
    • If you can still reproduce the bug set the STATUS to REOPENED and add a comment describing your testing
  • If you need help, send email to Anthony Hughes with the bug summary in the Subject field

Unconfirmed Bugs

  • Download and install the latest Firefox Nightly
  • Load this list of bug reports
  • Read each bug report until you find one you understand and want to test
  • Test the claim made in the bug report and see if you can reproduce it
  • If you can reproduce the bug, set the STATUS to NEW, add a comment describing your testing, and try to find a regression window
  • If you need help, send email to Anthony Hughes with the bug summary in the Subject field

Stability Bugs

  • Review the Firefox 28 Explosiveness Report
    • Any signatures rising significantly (>= 2.0 explosiveness rating) without a bug number should be reported to Bugzilla and nominated for tracking
    • If there is already a bug report, update the report with the latest numbers
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Ensure the bug is listed below using the bug number template
  • Review the Firefox 28 Topcrash Report
    • Any signature in the top 10 without a bug number should be reported to Bugzilla and nominated for tracking
    • If there is already a bug report, update the report with the latest ranking information
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Ensure the bug is listed below using the bug number template
  • Review the Firefox 28 tracked bugs that aren't tracked in 27 or 26.
    • Each bug listed needs to have its rank updated once per week
    • Use the topcrash reports from crash-stats to get the rank for each branch
    • Add this information to the bug report
    • Ensure the bug report has the topcrash-platform keyword added (where platform is the platform where crashes occur -- ie. topcrash-win for Windows)
    • Conversely, remove the topcrash keyword if the crash is no longer in the top-10
  • For each of these tasks, ensure the etherpad is updated with your name and the date checked

QA Wanted Bugs

  • Download and install the latest Firefox Nightly
  • Review this list of bug reports to find a bug you both understand and want to test (each should contain a comment detailing what's needed from QA)
    • If the bug needs steps to reproduce, try to find a reproducible set of steps
    • If the bug needs a regression window, try to find a regression window
    • If the bug is fixed and needs verification, test to confirm you can reproduce the bug on an older build but not the latest build
  • Any tested bugs should be added to the list below using the bug number template
  • Remove the qawanted keyword from any fulfilled request
  • If you need help, send email to Anthony Hughes with the bug summary in the Subject field

Aurora

The following tracks testing of Firefox 28 on the mozilla-aurora branch.

Daily Testing

Activity Description Last Checked Last Checked By
Automation Review the mozmill-daily testruns and report a bug for any failures and spikes in memory usage 2014-01-31 Anthony Hughes
Stability Report any explosive and high volume crashes, escalate tracked bugs, and update the etherpad 2014-01-30 Anthony Hughes
QA Wanted Review and investigate critical issues identified as needing QA assistance 2014-01-31 Anthony Hughes
Incoming/early triage Confirm and sort incoming bugs 2014-02-06 Liz Henry
Fix Triage Review the pushlog, uplifted, and landed bug fixes to flag for verification 2014-02-04 Ioana Budnar
Fix Verification Review and test the bugs flagged for verification, mark the bug verified if fixed or reopened if not 2014-02-04 Softvision team
Confirmations Review and test the unconfirmed bugs, mark reproduced bugs as NEW 2014-01-23 Ioana Budnar
Features Test and ensure Firefox features are signed-off without any blockers for Beta 2014-01-31 Cornel Ionce

Branch Migration

The following tracks sign-off of migration of Firefox 28 from the mozilla-central to the mozilla-aurora branch.

Tasks

  • [DONE] Updates verified throttled on aurora channel (QA)
  • [DONE] Set mozmill-aurora update tests to auroratest channel in MM-CI
  • [DONE] Automation daily testruns reviewed for failures
  • [DONE] Features qualified for uplift to Aurora by feature owners
  • [DONE] All flagged bug fixes verified
  • [DONE] All unconfirmed bugs triaged
  • [DONE] Check with Matt Wobensmith to flag Security concerns
  • [DONE] Topcrash reports reviewed for stability concerns
  • [DONE] Sign-off and verify updates on the aurora channel

Automated Tests

Testrun No. of Tests No. of Fails Failure Rate
Functional 50606 20 0.04%
Localization 456 38 8.33%
Endurance 1479 0 0.00%
Remote 14313 4 0.03%
Addons 32361 0 0.00%
Update 1064 80 7.52%

Feature Sign-offs

Confirm that all enabled features pass smoketests for primary user stories and that disabled features are disabled. Update the status of the feature below as follows:

  • SIGNED-OFF if the feature is enabled and passes all tests
  • DISABLED if the feature is disabled and verified disabled
  • FAILED if serious issues were encountered related to the feature

Additionally, sign-off any features not listed below tracked in this list

Feature Owner Status
Australis Cornel Ionce DISABLED
Shumway Bogdan Maris DISABLED
<input type=color> Paul Silaghi SIGNED-OFF
Disallow enumeration of navigator.plugins Catalin Varga SIGNED-OFF
Session Restore Improvements Cornel Ionce SIGNED-OFF
Notification center support for web notifications Bogdan Maris SIGNED-OFF
Manifest Editor Petruta Rasa SIGNED-OFF
VP9 video decoder Andrei Vaida SIGNED-OFF
Gamepad API Alexandra Lucinet SIGNED-OFF

Bug Fix Verifications

  • Triage this list of bugs
  • Attempt to reproduce the bug as described in a known broken build and the latest Aurora build
  • Update the bug with the results of your testing

Unconfirmed Bug Triage

Beta

The following tracks testing of Firefox 28 on the mozilla-beta branch.

Daily Testing

The following testing will be completed on a day-to-day basis with a goal to sign-off before the final Beta build.

Regression Testing

  • Run these tests for whatever platform(s) you have available
  • Report a bug if you encounter one
  • If you report a bug check to see if it's a regression by testing a previous Firefox version (if it doesn't reproduce in earlier versions it's a regression)
  • Add any reported bugs to the Tracked Issues section below
  • All results will appear in this dashboard

Stability Bugs

  • Familiarize yourself with the topcrash criteria
  • Review the explosiveness report and report anything with a rating >= 2
  • Review the topcrash report and report anything in the top-20
  • Review and update the tracked bugs with information from crash-stats
  • Update the crashkill etherpad with the status once completed
  • List any reported or unresolved topcrash bugs in the Top Crashes section above

QA Wanted Bugs

  • Review the critical issues for bugs needing QA assistance (check the backlog if that query is empty)
  • Complete the testing as requested and update the report with your details
  • If the request has been satisfied remove the qawanted keyword
  • If more information is required set the needinfo flag to whomever you need information from
  • List any bugs tested in the Tracked Bugs section below

Fix Verifications

  • Verify these bugs have been fixed before Release
  • Status: [ON TRACK] 31 bugs remaining

Fix Triage

  • Review the pushlog and flag bugs based on the criteria below
  • Review the uplifted fixes and flag bugs based on the criteria below
  • Review the landed fixes and flag bugs based on the criteria below
    • Any bugs flagged with the verifyme keyword will be tested in the next Beta
    • Any bugs flagged with the [qa-] whiteboard tag will not be tested
    • Any bugs flagged with the [good first verify] whiteboard tag will be tested by a volunteer
    • Any bugs flagged with the needinfo? flag will be verified by the reporter

Bug Confirmations

  • Review the unconfirmed bugs and test as needed
  • Any bugs you are able to reproduce should be marked as NEW
  • Any bugs you are unable to reproduce should be followed up with the bug reporter to provide more details
  • List any confirmed bugs in the section below

Beta 1

Build Info

Testing

  • [DONE] RC -> Beta Updates: verify updates from 27.0 to 28.0b1 work on betatest and releasetest
  • [DONE] Automation: review the functional, endurance, remote, add-ons tests and report bugs on any failures
  • [DONE] Smoketests: run smoketests to ensure scoped features uplifted successfully (Moztrap testrun)
  • [DONE] Locales testing: etherpad
  • [DONE] Fix Verifications: test and verify these bugs have been fixed in the current Beta and reopen if not
  • [DONE] Updates: run update tests on the betatest and releasetest channel and report any failures (reports missing due to bug 969052)
  • [DONE] Verify staged release notes (source)
  • [DONE] Live Updates: verify live updates work on the beta channel via automation and manual spotchecks (report)

RC to Beta Updates (betatest)

  • Run these Moztrap tests to ensure updates are working between Firefox 27b9 to 27rc to 28b1
  • Put the results in the following table
  • Engineering work is being tracked in bug 967965
Platform 27rc to 28b 27rc to 28b fallback 27b9 to 27rc to 28b 27b9 to 27rc to 28b fallback
Windows XP [DONE] en-US
[DONE] ach 		[DONE] en-US
[DONE] be 		[DONE] en-US
[DONE] ca 		[DONE] en-US
[DONE] da
Windows Vista [DONE] en-US
[DONE] es-AR 		[DONE] en-US
[DONE] fa 		[DONE] en-US
[DONE] ga-IE 		[DONE] en-US
[DONE] he
Windows 7 32-bit [DONE] en-US
[DONE] id 		[DONE] en-US
[DONE] ja 		[DONE] en-US
[DONE] kk 		[DONE] en-US
[DONE] lij
Windows 8.1 64-bit [DONE] en-US
[DONE] mk 		[DONE] en-US
[DONE] nl 		[DONE] en-US
[DONE] or 		[DONE] en-US
[DONE] pt-BR
Mac OSX 10.6 [DONE] en-US
[DONE] ro 		[DONE] en-US
[DONE] sl 		[DONE] en-US
[DONE] ta 		[DONE] en-US
[DONE] uk
Mac OSX 10.7 [DONE] en-US
[DONE] vi 		[DONE] en-US
[DONE] xh 		[DONE] en-US
[DONE] zh-CN 		[DONE] en-US
[DONE] af
Mac OSX 10.8 [DONE] en-US
[DONE] bg 		[DONE] en-US
[DONE] cs 		[DONE] en-US
[DONE] de 		[DONE] en-US
[DONE] en-ZA
Mac OSX 10.9 [DONE] en-US
[DONE] fr 		[DONE] en-US
[DONE] gu-IN 		[DONE] en-US
[DONE] hr 		[DONE] en-US
[DONE] it
Ubuntu 12.04 32-bit [DONE] en-US
[DONE] ko 		[DONE] en-US
[DONE] lt 		[DONE] en-US
[DONE] mr 		[DONE] en-US
[DONE] nn-NO
Ubuntu 13.10 64-bit [DONE] en-US
[DONE] pt-PT 		[DONE] en-US
[DONE] ru 		[DONE] en-US
[DONE] sk 		[DONE] en-US
[DONE] th

RC to Beta Updates (releasetest)

  • Run these Moztrap tests to ensure updates are working between Firefox 27b9 to 27rc to 28b1
  • Put the results in the following table, use the following reference build IDs to confirm your results
    • Firefox 27.0b9: 20140123185438
    • Firefox 27.0RC: 20140127194636
    • Firefox 28.0b1: 20140205162153

Results:

[DONE] Windows XP en-US, de, pt-BR Juan Becerra (:juanb)
[DONE] Windows 7 64-bit en-US, fr, zh-CN Stephen Donner (:stephend)
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, es-ES Tracy Walker (:tracy)
[DONE] Ubuntu 13.10 64-bit en-US, ru, pl Anthony Hughes (:ashughes)

Notes:

  • Mac OSX 27rc -> 28beta on releasetest did not require editing update-settings.ini despite not having "firefox-mozilla-beta" in the string

RC to Beta Updates (beta)

  • Run these Moztrap tests to ensure updates are working between Firefox 27rc to 28b1
  • Note: 27b9 -> 27rc -> 28b1 cannot be tested on live update channels
  • Put the results in the following table, use the following reference build IDs to confirm your results
    • Firefox 27.0RC: 20140127194636
    • Firefox 28.0b1: 20140205162153

Results:

[DONE] Windows XP en-US, de, pt-BR Anthony Hughes (:ashughes)
[DONE] Windows 7 64-bit en-US, fr, zh-CN Anthony Hughes (:ashughes)
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, es-ES Tracy Walker (:tracy)
[DONE] Ubuntu 13.10 64-bit en-US, ru, pl Anthony Hughes (:ashughes)

Bugs Found

  • bug 952034 Test failure 'Modal dialog has been found and processed' in /restartTests/testAddons_installFromFTP/test1.js
  • bug 967611 Failure in testSearchSuggestions.js: Suggestions from two search engines are available - '1' should equal '2'
  • bug 968108 Can't switch from Desktop to Metro mode with Firefox 28 beta 1 -- verified fixed
  • bug 968774 Metro interface is shown in a desktop window after crashing
  • bug 968809 VP9 videos that autoplay when opened are played the first 1-2 seconds at a lower luminosity.
  • bug 968844 Firefox 28 beta 1 closes when switching from Desktop to Metro mode
  • bug 968858 Download manager button fails to open the panel for multiple locales
  • bug 969052 On-demand Update results are not displaying in the Dashboard

Beta 2

Build Info

Testing

  • [DONE] Verify functional, endurance, remote, add-ons tests via automation
  • [DONE] Verify web compatibility and panning/scrolling via these regression tests
  • [DONE] Verify these bugs have been fixed in the current Beta
  • [DONE] Verify updates on the releasetest channel via automation
  • [DONE] Verify RC to Beta updates on the releasetest channel via manual spotcheck
  • [DONE] Verify updates on the beta channel via automation
  • [DONE] Verify RC to Beta updates on the beta channel via manual spotcheck

RC to Beta Updates (releasetest)

Verify Firefox 27.0rc builds update to the latest Beta on the releasetest channel using these steps:

  • Download and install Firefox 27.0
  • Locate channel-prefs.js in install folder and change the update channel
  • Locate the update-settings.ini file in the install folder and change ACCEPTED_MAR_CHANNEL_IDS to ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Open about:support to confirm you are using Firefox 28.0b2, based on the build ID

Results:

[DONE] Windows XP en-US, de, es-ES, pt-BR
[DONE] Windows 7 64-bit en-US, zh-CN, fr, it
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, pl, ro
[DONE] Ubuntu 13.10 64-bit en-US, ru, ko, hu

RC to Beta Updates (beta)

Verify Firefox 27.0rc builds update to the latest Beta on the beta channel using these steps:

  • Download and install Firefox 27.0
  • Locate channel-prefs.js in install folder and change the update channel
  • Locate the update-settings.ini file in the install folder and change ACCEPTED_MAR_CHANNEL_IDS to ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Open about:support to confirm you are using Firefox 28.0b2, based on the build ID

Results:

[DONE] Windows XP en-US, de, es-ES, pt-BR
[DONE] Windows 7 64-bit en-US, zh-CN, fr, it
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, pl, ro
[DONE] Ubuntu 13.10 64-bit en-US, ru, ko, hu

Bugs Found

  • bug 971007 Failure in /testGeolocation/testShareLocation.js: "Geolocation position is: Unknown error acquiring position"
  • bug 964191 Crash [@ gdk_visual_get_blue_pixel_details ] under PK11PasswordPrompt during mozmill testrun

Beta 3

Build Info

Testing

  • [DONE] Verify functional, endurance, remote, add-ons tests via automation
  • [DONE] Regression testing Geolocation
  • [DONE] Verify these bugs have been fixed in the current Beta
  • [DONE] Verify updates on the releasetest channel via automation
  • [DONE] Verify RC to Beta updates on the releasetest channel via manual spotcheck
  • [DONE] Verify updates on the beta channel via automation
  • [DONE] Verify RC to Beta updates on the beta channel via manual spotcheck

RC to Beta Updates (releasetest)

Verify Firefox 27.0rc builds update to the latest Beta on the releasetest channel using these steps:

  • Download and install Firefox 27.0
  • Locate channel-prefs.js in install folder and change the update channel
  • Locate the update-settings.ini file in the install folder and change ACCEPTED_MAR_CHANNEL_IDS to ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Open about:support to confirm your build ID matches the one above in the Build Info section

Results:

[DONE] Windows XP en-US, de, es-ES, pt-BR
[DONE] Windows 7 64-bit en-US, zh-CN, fr, it
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, pl, ro
[DONE] Ubuntu 13.10 64-bit en-US, ru, ko, hu

RC to Beta Updates (beta)

Verify Firefox 27.0rc builds update to the latest Beta on the beta channel using these steps:

  • Download and install Firefox 27.0
  • Locate channel-prefs.js in install folder and change the update channel
  • Locate the update-settings.ini file in the install folder and change ACCEPTED_MAR_CHANNEL_IDS to ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Open about:support to confirm you are using Firefox 28.0b2, based on the build ID

Results:

[DONE] Windows XP en-US, de, es-ES, pt-BR
[DONE] Windows 7 64-bit en-US, zh-CN, fr, it
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, pl, ro
[DONE] Ubuntu 13.10 64-bit en-US, ru, ko, hu

Issues

  • bug 972959 [zh-CN] Failures in /restartTests/testAddons_enableDisableExtension
  • bug 951244 [ga-IE] Test failure 'TabView is not open.' in /testTabView/testToggleTabView.js and /testTabView/testTabGroupNaming.js
  • bug 942737 Test failure "Suggestions from two search engines are available - '1' should equal '2' " in testSearchSuggestions.js

Beta 4

Build Info

Testing

  • [DONE] Verify functional, endurance, remote, add-ons tests via automation
  • [DONE] Verify Malay [ms] builds are working via automation and manual smoketesting (report)
  • [DONE] Verify these bugs have been fixed in the current Beta
  • [DONE] Verify updates on the releasetest channel via automation
  • [DONE] Verify RC to Beta updates on the releasetest channel via manual spotcheck
  • [DONE] Verify updates on the beta channel via automation
  • [DONE] Verify RC to Beta updates on the beta channel via manual spotcheck

RC to Beta Updates (releasetest)

Verify Firefox 27.0rc builds update to the latest Beta on the releasetest channel using these steps:

  • Download and install Firefox 27.0
  • Locate channel-prefs.js in install folder and change the update channel
  • Locate the update-settings.ini file in the install folder and change ACCEPTED_MAR_CHANNEL_IDS to ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Open about:support to confirm you are using the latest Beta based on the build ID

Results:

[DONE] Windows XP en-US, de, es-ES, pt-BR
[DONE] Windows 7 64-bit en-US, zh-CN, fr, it
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, pl, ro
[DONE] Ubuntu 13.10 64-bit en-US, ru, ko, hu

RC to Beta Updates (beta)

Verify Firefox 27.0rc builds update to the latest Beta on the beta channel using these steps:

  • Download and install Firefox 27.0
  • Locate channel-prefs.js in install folder and change the update channel
  • Locate the update-settings.ini file in the install folder and change ACCEPTED_MAR_CHANNEL_IDS to ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Open about:support to confirm you are using the latest Beta based on the build ID

Results:

[DONE] Windows XP en-US, de, es-ES, pt-BR
[DONE] Windows 7 64-bit en-US, zh-CN, fr, it
[DONE] Mac OSX 10.9 en-US, ja-JP-mac, pl, ro
[DONE] Ubuntu 13.10 64-bit en-US, ru, ko, hu

Issues

Mozmill Test Failures

  • bug 974586 Failure in /testLayout/testNavigateFTP.js - Timeout waiting for page loaded
  • bug 974582 Failure in /testGeolocation/testShareLocation.js: Unknown error acquiring position
  • bug 972433 Test failure "AddonsManager_isAddonEnabled: Add-on has been specified
  • bug 952034 Test failure 'Modal dialog has been found and processed' in /restartTests/testAddons_installFromFTP/test1.js
  • bug 951244 [ga-IE] Test failure 'TabView is not open.' in /testTabView/testToggleTabView.js and /testTabView/testTabGroupNaming.js
  • bug 950025 Test failure "Blocklist has been updated." in testAddons_installUninstallSoftBlocklistedExtension/test4.js
  • bug 942737 Test failure "Suggestions from two search engines are available"
  • bug 922087 Test failure "The tab has loaded" in /testSearch/testSearchSelection.js

Manually Reproduced Bugs

  • bug 942019 SessionStore doesn't take Windows DPI settings into account when restoring window position
  • bug 929067 Canceling a download from the Downloads Panel sometimes pauses it instead

Beta 5

No Beta 5 was shipped due to a shortend week and taking one extra day to ship Firefox 28.0b4 due to prior stability concerns.

Beta 6

Build Info

Testing

  • [DONE] Verify functional, endurance, remote, add-ons tests via automation
  • [DONE] Verify these bugs have been fixed in the current Beta
  • [DONE] Verify updates on the releasetest channel via automation
  • [DONE] Verify RC to Beta updates on the releasetest channel via manual spotcheck
  • [DONE] Verify updates on the beta channel via automation
  • [DONE] Verify RC to Beta updates on the beta channel via manual spotcheck

RC to Beta Updates (releasetest)

Verify Firefox 27.0rc builds update to the latest Beta on the releasetest channel using these steps:

  • Download and install Firefox 27.0
  • Change "release" to "releasetest" in %install_folder%/defaults/pref/channel-prefs.js
  • Change "firefox-mozilla-release" to "firefox-mozilla-beta,firefox-mozilla-release" in %install_folder%/update-settings.ini
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Reopen the About dialog to confirm you're using Firefox 28 on the releasetest channel and that you don't get another update
  • Open about:support to confirm the build ID matches the ID for this Beta

Results:

[DONE] Windows XP en-US, de
[DONE] Mac OSX 10.9 en-US, ja-JP-mac
[DONE] Ubuntu 13.10 64-bit en-US, pt-BR

RC to Beta Updates (beta)

Verify Firefox 27.0rc builds update to the latest Beta on the beta channel using these steps:

  • Download and install Firefox 27.0
  • Locate channel-prefs.js in install folder and change the update channel
  • Locate the update-settings.ini file in the install folder and change ACCEPTED_MAR_CHANNEL_IDS to ACCEPTED_MAR_CHANNEL_IDS=firefox-mozilla-beta,firefox-mozilla-release
  • Start Firefox and check for updates through the About Firefox dialog
  • A complete update should be found and installed
  • Restart Firefox when prompted
  • Open about:support to confirm you are using the latest Beta based on the build ID

Results:

[DONE] Windows XP en-US, de
[DONE] Mac OSX 10.9 en-US, ja-JP-mac
[DONE] Ubuntu 13.10 64-bit en-US, pt-BR

Issues

Mozmill Test Failures

  • bug 974892 [ms] Failure in /testSearch/testSearchSuggestions.js: Suggestions from two search engines are available
  • bug 958068 Failure in /testPrivateBrowsing/testFlashCookie.js: Cookie value is displayed in private mode
  • bug 952034 Failure in /restartTests/testAddons_installFromFTP: Modal dialog has been found and processed
  • bug 942737 Failure in /testSearch/testSearchSuggestions.js: Suggestions from two search engines are available

Manually Reproduced Bugs

  • no bugs reported

Beta 7

Build Information

Automation

Manual Tests

Software Updates on the releasetest channel:

  • RC to Beta Updates
    • [DONE] Windows XP: en-US, de
    • [DONE] Mac OSX 10.7: en-US, ja-JP-mac
    • [DONE] Ubuntu 13.10 64-bit: en-US, pt-BR
    • [DONE] Windows 8 Metro: en-US, fr
  • Metro Firefox Updates
    • [DONE] Windows 8.1 w/Firefox 28.0b6: en-US, es-ES
    • [DONE] Windows 8.1 w/Firefox 28.0b3: en-US, zh-TW

Software Updates on the beta channel:

  • RC to Beta Updates
    • [DONE] Windows XP: en-US, de
    • [DONE] Mac OSX 10.7: en-US, ja-JP-mac
    • [DONE] Ubuntu 13.10 64-bit: en-US, pt-BR
    • [DONE] Windows 8 Metro: en-US, fr
  • Metro Firefox Updates
    • [DONE] Windows 8.1 w/Firefox 28.0b6: en-US, es-ES
    • [DONE] Windows 8.1 w/Firefox 28.0b3: en-US, zh-TW

Issues

  • no bugs reported

Beta 8

Build Information

Automation

Manual Tests

Metro Firefox Regression Tests:

  • list tests performed and devices tested

Software Updates on the releasetest channel:

  • RC to Beta Updates
    • [DONE] Windows XP: en-US, de
    • [DONE] Mac OSX 10.9: en-US, ja-JP-mac
    • [MISSED] - no machine access Ubuntu 13.10 64-bit: en-US, zh-TW
    • [DONE] Windows 8 Metro: en-US, es-ES
  • Metro Firefox Updates
    • [DONE] Windows 8.1 w/Firefox 28.0b7: en-US, it
    • [DONE] Windows 8.1 w/Firefox 28.0b4: en-US, fr

Software Updates on the beta channel:

  • RC to Beta Updates
    • [DONE] Windows XP: en-US, de
    • [DONE] Mac OSX 10.9: en-US, ja-JP-mac
    • [MISSED] - no machine access Ubuntu 13.10 64-bit: en-US, zh-TW
    • [DONE] Windows 8 Metro: en-US, es-ES
  • Metro Firefox Updates
    • [DONE] Windows 8.1 w/Firefox 28.0b7: en-US, it
    • [DONE] Windows 8.1 w/Firefox 28.0b4: en-US, fr

Issues

  • no bugs reported

Beta 9

Build Information

Automation

Manual Tests

Metro Firefox Regression Tests:

  • list tests performed and devices tested

Software Updates on the releasetest channel:

  • RC to Beta Updates
    • [DONE] Windows XP: en-US, de
    • [DONE] Mac OSX 10.9: en-US, ja-JP-mac
    • [DONE] Ubuntu 13.10 64-bit: en-US, zh-TW
    • [DONE] Windows 8 Metro: en-US, es-ES
  • Metro Firefox Updates
    • [DONE] Windows 8.1 w/Firefox 28.0b7: en-US, it
    • [DONE] Windows 8.1 w/Firefox 28.0b4: en-US, fr

Software Updates on the beta channel:

  • RC to Beta Updates
    • [DONE] Windows XP: en-US, de
    • [DONE] Mac OSX 10.9: en-US, ja-JP-mac
    • [DONE] Ubuntu 13.10 64-bit: en-US, zh-TW
    • [DONE] Windows 8 Metro: en-US, es-ES
  • Metro Firefox Updates
    • [DONE] Windows 8.1 w/Firefox 28.0b7: en-US, it
    • [DONE] Windows 8.1 w/Firefox 28.0b4: en-US, fr

Issues

  • no bugs reported

Release

Release Candidate 1

Build Information

Automation

Tests Windows Mac OS X Ubuntu Linux
functional (config) [DONE] [DONE] [DONE]
endurance [DONE] [DONE] [DONE]
remote [DONE] [DONE] [DONE]
add-ons [DONE] [DONE] [DONE]
releasetest updates (config) [DONE] [DONE] [DONE]
beta updates (config) [BLOCKED] [BLOCKED] [BLOCKED]

Manual Tests

Testing Windows XP Windows 8.1 Mac OSX 10.9 Ubuntu 13.10
Metro n/a [BLOCKED] n/a n/a
VP9 [DONE] [DONE] [DONE] [DONE]
WebVTT [DONE] [DONE] [DONE] [DONE]
Session Restore [DONE] [DONE] [DONE] [DONE]
Plugins (Flash, Java, Silverlight, etc) [DONE] [DONE] [DONE] [DONE]
Partner updates on releasetest channel, general functionality, branding, etc [DONE] [DONE] [DONE] [DONE]
MS/SW Betas update to 28.0b9 on releasetest, not RC [DONE] 28.0b8 ms
[DONE] 28.0b7 sw 		[DONE] 28.0b7 ms
[DONE] 28.0b6 sw 		[DONE] 28.0b6 ms
[DONE] 28.0b4 sw 		[DONE] 28.0b4 ms
[DONE] 28.0b3 sw
27.0 -> 28.0rc updates on releasetest [DONE] en-US (direct)
[DONE] de (fallback) 		[DONE] en-US (fallback)
[DONE] fr (direct) 		[DONE] en-US (direct)
[DONE] ja-JP-mac (fallback) 		[DONE] en-US (fallback)
[DONE] zh-TW (direct)

Issues

  • Some Metro related issues reported (details)

Release Candidate 2

Build Information

Automated Tests

Bug Fix Verifications

Issues

  • no new issues reported

Safebrowsing Hotfix

bug 985627 temporarily turn off malware blocklist and allowlists

Staging

Installations Tested:

  • Firefox 27.0
  • Firefox 27.0.1
  • Firefox 28.0
  • Firefox 27.0 updated to 28.0 after hotfix installed
  • Firefox 27.0.1 updated to 28.0 before hotfix installed
  • Firefox 28.0 paved over Firefox 27.0 after hotfix installed
  • Firefox 27.0 paved over Firefox 28.0 before hotfix installed
  • Firefox 26.0 => no hotfix installed

Checks Performed:

  • hotfix is installed after a background ping to addons-dev.allizom.org
    • urlclassifier.download_allow_table value is wiped
    • ulrclassifier.download_block_table value is wiped
  • wait 45 minutes after the hotfix is installed
    • goog-phish-shavar* and goog-malware-shavar* files updated
    • goog-badbinurl-shavar and goog-downloadwhite-digest256 files should not exist
  • restart to Firefox to confirm removal of hotfix add-on

Result: ALL PASS

Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_28/Test_Plan&oldid=953547"