Releases/Firefox 14/Test Plan

From MozillaWiki
Jump to: navigation, search

« Firefox 13 | Firefox 14 | Firefox 15 »

Contents

Firefox 14 Test Plan

Summary

Milestone Firefox 14
Status Currently in Beta -- Shipping July 17, 2012. Tentative.
Lead Juan Becerra (irc: juanb)
2nd Anthony Hughes (irc: ashughes)
Builds [ftp] [http] [web]

The following is the test plan for Firefox 14 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 14. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

  • 2012-03-13: [DONE] Nightly Merge
  • 2012-04-24: [DONE] Aurora Merge
  • 2012-06-05: [DONE] Pre-Beta feature sign-offs
  • 2012-06-07: [DONE] Beta 6 (Starting at version 6 to sync with Firefox Mobile)
  • 2012-06-14: [DONE] Beta 7
  • 2012-06-21: [DONE] Beta 8
  • 2012-06-28: [DONE] Beta 9
  • 2012-07-05: [DONE] Beta 10
  • 2012-07-11: [DONE] Beta 11
  • 2012-07-13: [ON TRACK] Release Candidate
  • 2012-07-17: {{|}} Final

Features

See the Feature Release Tracking wiki for a general overview

[DONE]
[ON TRACK]
[AT RISK]


[MISSED]

Nightly Merge

Functional Automation - result
Update Spotchecks - result

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do
  1. Check the Endurance Charts
  2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
  3. If the event happens twice in a row, file a bug under Mozmill Tests
  4. Notify Juan Becerra of the issue
  5. Record the event in this wiki below
Current Status (Mihaela Velimiroviciu) - Last Updated
2012-06-25
  • bug 752887 - Increased memory usage for endurance tests on Linux Aurora branch
  • bug 768026 - Higher memory consumption on Win 6.0 on Beta branch since the merge

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do
  1. Check the testrun reports
  2. If you see any failing tests, check that there are not any bugs on file for it already
  3. If not, file a bug under Mozmill Tests
  4. Notify Juan Becerra of the issue
  5. Record the event in this wiki below
Current Status (Vlad Maniac / Alex Lakatos) - Last Updated
  • No issues found

Unconfirmed Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs. The below charts provides a summary of the unconfirmed bugs currently in the Firefox product:

P1 Task - Get UNCONFIRMED Bugs Out of Firefox General and Untriaged If Possible
  1. Check this <insert link> for UNCONFIRMED bugs found for this Firefox Release's Beta Timeframe
  2. Read each bug and determine what product and component they belong under in bugzilla
  3. Move the bug to the right product and component
  4. CC any applicable team members that may be interested in the bug
P2 Task - Confirm UNCONFIRMED Bugs in this 6-Week Cycle
  1. Check the <insert link> bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Juan Becerra of the NEW issue
P3 Task - Confirm UNCONFIRMED Bugs from Other Releases
  1. Check the UNCONFIRMED bugs
  2. Test the bugs in the latest Firefox Nightly
  3. If you can't reproduce the bug, ask for the reporter to provide more information
  4. If you reproduce the bug, add your testing information and mark the bug NEW
  5. Notify Juan Becerra of the NEW issue
Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet

Aurora Merge

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-04-26
Windows Vista PASS - 2012-04-26
Windows 7 32-bit PASS - 2012-04-26
Windows 7 64-bit PASS - 2012-04-26
Mac OS X 10.6 PASS - 2012-04-26
Mac OS X 10.7 PASS - 2012-04-26
Ubuntu 11.10 32-bit PASS - 2012-04-26
Ubuntu 11.10 64-bit PASS - 2012-04-26
Test Failures
  • /testPasswordManager/testPasswordNotificationBar.js (bug 749156)
  • /testPasswordManager/testPasswordSavedAndDeleted.js (bug 749206)
  • /testSecurity/testBlueLarry.js (bug 748686)
  • /testSecurity/testGreenLarry.js (bug 748686)
  • /testSecurity/testGreyLarry.js (bug 748686)

Updates

The following are details of update verifications for Firefox 12 merge to Aurora

Auroratest
  • [ON TRACK] [ashughes] Windows XP:
  • [ON TRACK] [ashughes] Windows 7:
  • [ON TRACK] [ashughes] Mac OS X 10.6:
  • [ON TRACK] [ashughes] Ubuntu 11.04:
Aurora
  • [ON TRACK] [ashughes] Windows XP:
  • [ON TRACK] [ashughes] Windows 7:
  • [ON TRACK] [ashughes] Mac OS X 10.6:
  • [ON TRACK] [ashughes] Ubuntu 11.04:
Issues Found

Manual Testing

Smoketests

Run the Aurora Merge Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-04-27
Windows Vista Simona PASS - 2012-04-27
Windows 7 32-bit Paul PASS - 2012-04-27
Windows 7 64-bit Simona PASS - 2012-04-27
Mac OS X 10.6.8 Paul PASS - 2012-04-27
Mac OS X 10.7.3 Paul PASS - 2012-04-27
Ubuntu 11.10 32-bit Virgil PASS - 2012-04-27
Ubuntu 11.10 64-bit Mihaela PASS - 2012-04-27

Results in Litmus

Feature Sign-offs

Run the Aurora 14.0 sign-offs tests in Litmus and report results below.

Feature Owner Win32 Mac Linux
Panel-based Download Manager Simona Badau PASS - 2012-04-27 PASS - 2012-04-27 PASS - 2012-04-27
Windows 8 Metro Firefox Vlad Ghetiu N/A N/A N/A
Cycle Collector Perf improvements Virgil Dicu PASS - 2012-04-27 PASS - 2012-04-30 PASS - 2012-04-30
Incremental Garbage Collection Ioana Budnar PASS - 2012-04-27 PASS - 2012-04-27 PASS - 2012-04-27
Hang Detector and Reporter Simona Badau N/A N/A N/A
URL Autocomplete Vlad Ghetiu PASS - 2012-04-27 PASS - 2012-04-27 PASS - 2012-04-27
Opt-in activation for plugins Paul Silaghi PASS - 2012-04-27 PASS - 2012-04-30 PASS - 2012-04-30
Show PDF Inline Mihaela Velimiroviciu N/A N/A N/A
Firefox migrators Paul Silaghi PASS - 2012-04-30 PASS - 2012-04-30 N/A
Silent Update: Background updates Vlad Ghetiu N/A N/A N/A
Web Apps integration Jason Smith FAIL - 2012-05-01 FAIL - 2012-05-01 N/A - 2012-05-01
Pseudo-Class Lock Ioana Budnar PASS - 2012-04-30 PASS - 2012-04-30 PASS - 2012-04-30
Support for Mouse Capture / Mouse Lock Virgil Dicu FAIL - 2012-04-30 FAIL - 2012-04-30 FAIL - 2012-04-30

Results in Litmus


Notes:

Issues Found

Full Query
ID Summary Priority Status
749579 Visual glitch when playing chrome angrybirds in SD -- RESOLVED
750245 Mouse lock pointer not working for quake demo -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
564934 Implement new Download Manager UI for browser -- RESOLVED
726447 [meta] Theming and visual design -- NEW
726453 [meta] Improve accessibility and keyboard interaction -- NEW
728692 Javascript becomes slow after Incremental GC gets disabled -- RESOLVED
735183 No Home Page Selection screen on first run import wizard -- RESOLVED
735944 about:support says "incremental gc: 1" when javascript.options.mem.gc_incremental is disabled -- VERIFIED
743102 Desktop Click to play blocks Silverlight - prompted to Install -- VERIFIED
745187 Click-to-activate plugins which are removed immediately after adding don't trigger the plugin notification (Google Earth Plugin doesn't work with click-to-play enabled) P1 VERIFIED
746658 Failed downloads can't be removed from panel -- VERIFIED
746739 The new download list (when downloads button is clicked) looks bad -- RESOLVED
746743 The new download list doesn't scroll automatically to the newly started download -- RESOLVED
746751 Download panel icon doesn't appear -- RESOLVED
746760 Tooltip covers part of the download list -- NEW
746772 Downloads button disappears from toolbar when customizing -- VERIFIED
746783 "Open Containing Folder" doesn't appear in context menu of downloads P2 RESOLVED
747310 Custom toolbar disappears when Edit Bookmarks Panel or Downloads Panel pop up. -- RESOLVED
747909 Youtube.com In Full Screen mode - copy actions are not allowed on HTML5 videos -- RESOLVED
748381 Downloads indicator should persist in the toolbar (not auto-hide) -- VERIFIED

18 Total; 3 Open (16.67%); 9 Resolved (50%); 6 Verified (33.33%);


Pre-beta features sign offs

Run the Firefox 14.0 pre-beta sign off tests in Litmus and report results below.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu PASS - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31
Opt-in activation for plugins Paul Silaghi PASS - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31
Firefox migrators - Safari Migrator Paul Silaghi PASS - 2012-05-31 PASS - 2012-05-31 N/A - 2012-05-31
Pseudo-Class Lock Ioana Budnar FAIL - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31
Support for Mouse Capture / Mouse Lock Virgil Dicu PASS - 2012-05-31 PASS - 2012-05-31 PASS - 2012-05-31

Results in Litmus

Issues Found

Full Query
ID Summary Priority Status
760171 The inspector node menu feature allows the user to remove tabbed browsing P1 RESOLVED
760175 The :focus pseudoClass is not kept for the parent node when deleting a node -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Known Issues

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Beta 6

Builds FTP, HTTP
ID 20120605113340
Changeset 89ec8943347a
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [DONE] Get final sign-off from Bob Moss
  • [DONE] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-06
Windows Vista PASS - 2012-06-06
Windows 7 32-bit PASS - 2012-06-06
Windows 7 64-bit PASS - 2012-06-06
Mac OS X 10.6 PASS - 2012-06-06
Mac OS X 10.7 PASS - 2012-06-06
Ubuntu 11.10 32-bit PASS - 2012-06-06
Ubuntu 11.10 64-bit PASS - 2012-06-06
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

  • Working on cloning the branch as of 06-04 @11:50pm Pacific
Platform Assignee Result
Windows XP Simona PASS - 2012-06-06
Windows Vista tester RESULT - 2012-06-06
Windows 7 32-bit tester RESULT - 2012-06-06
Windows 7 64-bit tester RESULT - 2012-06-06
Mac OS X 10.6.8 tester RESULT - 2012-06-06
Mac OS X 10.7.4 tester RESULT - 2012-06-06
Ubuntu 12.04 32-bit tester RESULT - 2012-06-06
Ubuntu 11.10 64-bit tester RESULT - 2012-06-06
Windows 7 32-bit (es-AR) Gabriela PASS - 2012-06-06

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit tester RESULT - 2012-06-06
Mac OS X 10.6.8 tester RESULT - 2012-06-06
Ubuntu 11.10 64-bit tester RESULT - 2012-06-06
Windows XP tester RESULT - 2012-06-06
Windows 7 32-bit (es-AR) Gabriela PASS - 2012-06-07

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
  2. [DONE] Exploratory testing on the new features Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Opt-in activation for plugins Paul Silaghi PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Firefox migrators - Safari Migrator Paul Silaghi PASS - 2012-06-06 PASS - 2012-06-06 N/A
Pseudo-Class Lock Ioana Budnar PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Support for Mouse Capture / Mouse Lock Virgil Dicu PASS - 2012-06-06 PASS - 2012-06-06 PASS - 2012-06-06
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
  • Open a terminal window and cd to the folder Firefox.app is located
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test.

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-06 RESULT - 2012-06-06
Windows Vista PASS - 2012-06-06 RESULT - 2012-06-06
Windows 7 32-bit PASS - 2012-06-06 RESULT - 2012-06-06
Windows 7 64-bit PASS - 2012-06-06 RESULT - 2012-06-06
Mac OS X 10.6 PASS - 2012-06-06 RESULT - 2012-06-06
Mac OS 10.7 PASS - 2012-06-06 RESULT - 2012-06-06
Ubuntu 11.10 32-bit PASS - 2012-06-06 RESULT - 2012-06-06
Ubuntu 11.10 64-bit PASS - 2012-06-06 RESULT - 2012-06-06
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 14 25 36%
QA+ SG: for the Security team to verify security fixes 7 23 23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Issues Found

Full Query
ID Summary Priority Status
761990 On Windows the "Feedback" button's dropdown arrow overlaps the label -- RESOLVED
762011 [highlighter] Node operation bar displayed but not accessible for on mouse hover elements -- RESOLVED
762456 In full screen - Flash player behaves differently if opened from the button or context menu -- RESOLVED
762483 On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta -- RESOLVED

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
675645 Firefox freezes when using Pop out while in Fullscreen mode on Youtube P3 RESOLVED
711742 HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video -- RESOLVED
747858 A horizontal line is displayed on the Play button from HTML5 videos -- RESOLVED
754829 In Full Screen mode - YouTube videos player changes dimmension -- RESOLVED
760171 The inspector node menu feature allows the user to remove tabbed browsing P1 RESOLVED
760175 The :focus pseudoClass is not kept for the parent node when deleting a node -- RESOLVED

6 Total; 0 Open (0%); 6 Resolved (100%); 0 Verified (0%);


Beta 6 Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 7

Builds FTP, HTTP
ID TBD
Changeset 89ec8943347a
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-12
Windows Vista PASS - 2012-06-12
Windows 7 32-bit PASS - 2012-06-12
Windows 7 64-bit PASS - 2012-06-12
Mac OS X 10.6 PASS - 2012-06-12
Mac OS X 10.7 PASS - 2012-06-12
Ubuntu 11.10 32-bit PASS - 2012-06-12
Ubuntu 11.10 64-bit PASS - 2012-06-12
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP tester RESULT - 2012-06-13
Windows Vista tester RESULT - 2012-06-13
Windows 7 32-bit Simona In progress - 2012-06-13
Windows 7 64-bit tester RESULT - 2012-06-13
Mac OS X 10.6.8 VladG PASS - 2012-06-13
Mac OS X 10.7.4 tester RESULT - 2012-06-13
Ubuntu 12.04 32-bit tester RESULT - 2012-06-13
Ubuntu 11.10 64-bit tester RESULT - 2012-06-13
Windows 7 32-bit (es-AR) Gabriela RESULT - 2012-06-13

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit tester RESULT - 2012-06-12
Mac OS X 10.6.8 tester RESULT - 2012-06-12
Ubuntu 11.10 64-bit tester RESULT - 2012-06-12
Windows XP tester RESULT - 2012-06-12
Windows 7 32-bit (es-AR) Gabriela PASS - 2012-06-07

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
  2. [DONE] Exploratory testing on the new features Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Opt-in activation for plugins Paul Silaghi RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-06-12 RESULT - 2012-06-12 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-06-12 RESULT - 2012-06-12 RESULT - 2012-06-12
Mac OS 10.8 Signing Required n/a RESULT - 2012-06-12 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - the output is containing "com.apple.FinderInfo" and "com.apple.quarantine"
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG

Spot check build after this test.

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-12 RESULT - 2012-06-12
Windows Vista PASS - 2012-06-12 RESULT - 2012-06-12
Windows 7 32-bit PASS - 2012-06-12 RESULT - 2012-06-12
Windows 7 64-bit PASS - 2012-06-12 RESULT - 2012-06-12
Mac OS X 10.6 PASS - 2012-06-12 RESULT - 2012-06-12
Mac OS 10.7 PASS - 2012-06-12 RESULT - 2012-06-12
Ubuntu 11.10 32-bit PASS - 2012-06-12 RESULT - 2012-06-12
Ubuntu 11.10 64-bit PASS - 2012-06-12 RESULT - 2012-06-12
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 20 19 51%
QA+ SG: for the Security team to verify security fixes 7 23 23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Need monitoring of crash stats for bug 760190
  • bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
  • bug 761990 Feedback button regression - investigated and updated (Simona).
  • bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
  • bug 748569 needs verification (migration code)
  • Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
  • bug 755728 low priority investigation - investigated and updated (Paul).

Issues Found

Full Query
ID Summary Priority Status
764366 When logging into a site with multiple accounts and only saving the passwords for some of them, not all emails/usernames are remembered -- RESOLVED
764385 [10.7][10.8]Silverlight keeps on crashing on Firefox 14 beta -- RESOLVED
764388 "http://www." is appended to the string entered in the location bar, when it contains at least one dot , even if the string after the dot is obviously not a domain name -- RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
587874 Undo Close Tab should only open tabs from the current group P1 RESOLVED
675645 Firefox freezes when using Pop out while in Fullscreen mode on Youtube P3 RESOLVED
711742 HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video -- RESOLVED
747858 A horizontal line is displayed on the Play button from HTML5 videos -- RESOLVED
754829 In Full Screen mode - YouTube videos player changes dimmension -- RESOLVED
760171 The inspector node menu feature allows the user to remove tabbed browsing P1 RESOLVED
760175 The :focus pseudoClass is not kept for the parent node when deleting a node -- RESOLVED
761990 On Windows the "Feedback" button's dropdown arrow overlaps the label -- RESOLVED
762011 [highlighter] Node operation bar displayed but not accessible for on mouse hover elements -- RESOLVED
762456 In full screen - Flash player behaves differently if opened from the button or context menu -- RESOLVED
762483 On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta -- RESOLVED
764062 Autofilled suggestions in address bar do not include port numbers P3 RESOLVED

12 Total; 0 Open (0%); 12 Resolved (100%); 0 Verified (0%);


Beta 7 Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 8

Builds FTP, HTTP
ID 20120619191901
Changeset Changeset
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-20
Windows Vista PASS - 2012-06-20
Windows 7 32-bit PASS - 2012-06-20
Windows 7 64-bit PASS - 2012-06-20
Mac OS X 10.6 PASS - 2012-06-20
Mac OS X 10.7 PASS - 2012-06-20
Ubuntu 11.10 32-bit PASS - 2012-06-20
Ubuntu 11.10 64-bit PASS - 2012-06-20
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Simona PASS - 2012-06-19
Windows Vista tester RESULT - 2012-06-19
Windows 7 32-bit tester RESULT - 2012-06-19
Windows 7 64-bit tester RESULT - 2012-06-19
Mac OS X 10.6.8 tester RESULT - 2012-06-19
Mac OS X 10.7.4 tester RESULT - 2012-06-19
Ubuntu 12.04 32-bit tester RESULT - 2012-06-19
Ubuntu 11.10 64-bit tester RESULT - 2012-06-19
Windows 8 32-bit build (en-US) juanb PASS - 2012-06-20

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform Assignee Result: report
Windows 7 32-bit tester RESULT - 2012-06-19
Mac OS X 10.6.8 tester RESULT - 2012-06-19
Ubuntu 11.10 64-bit tester RESULT - 2012-06-19
Windows XP tester RESULT - 2012-06-19

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, addons, bookmarks, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Opt-in activation for plugins Paul Silaghi RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-06-19 RESULT - 2012-06-19 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-06-19 RESULT - 2012-06-19 RESULT - 2012-06-19
Mac OS 10.8 Signing Required n/a RESULT - 2012-06-19 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS (Ioana)
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS (Ioana)
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS (Ioana)
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS (Ioana)
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS (Ioana)
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS (Ioana)
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS (Ioana)

Spot check build after this test. Results

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP RESULT - 2012-06-19 RESULT - 2012-06-19
Windows Vista RESULT - 2012-06-19 RESULT - 2012-06-19
Windows 7 32-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Windows 7 64-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Mac OS X 10.6 RESULT - 2012-06-19 RESULT - 2012-06-19
Mac OS 10.7 RESULT - 2012-06-19 RESULT - 2012-06-19
Ubuntu 11.10 32-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Ubuntu 11.10 64-bit RESULT - 2012-06-19 RESULT - 2012-06-19
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 7 32 89%
SG with testcase: for the Security team to verify security fixes 7 23 23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs 297 0 100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Need monitoring of crash stats for bug 760190
  • bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
  • bug 761990 Feedback button regression - investigated and updated (Simona).
  • bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
  • bug 748569 needs verification (migration code)
  • Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
  • bug 755728 low priority investigation - investigated and updated (Paul).
  • bug 764157 regression range wanted - found regression range/updated (Virgil)
  • bug 754000 performance regression - investigated and updated (Paul).

Issues Found

Full Query
ID Summary Priority Status
766548 [adbe 3226428] Taking a speed test on YouTube hangs Firefox -- RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
675645 Firefox freezes when using Pop out while in Fullscreen mode on Youtube P3 RESOLVED
711742 HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video -- RESOLVED
747858 A horizontal line is displayed on the Play button from HTML5 videos -- RESOLVED
754829 In Full Screen mode - YouTube videos player changes dimmension -- RESOLVED
760171 The inspector node menu feature allows the user to remove tabbed browsing P1 RESOLVED
760175 The :focus pseudoClass is not kept for the parent node when deleting a node -- RESOLVED
761990 On Windows the "Feedback" button's dropdown arrow overlaps the label -- RESOLVED
762011 [highlighter] Node operation bar displayed but not accessible for on mouse hover elements -- RESOLVED
762456 In full screen - Flash player behaves differently if opened from the button or context menu -- RESOLVED
762483 On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta -- RESOLVED

10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);


Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 9

NOTE this is a chemspill Beta to mitigate requirement to restart OS on Flash update
Builds FTP, HTTP
ID 20120624012213
Changeset d050090e578c
Checklist
  • [DONE] Functional automated tests
  • [ON TRACK] Manual tests
  • [ON TRACK] Bug fix verifications
  • [DONE] Update tests on releasetest
  • {{ |}} Peer sign-off
  • {{ |}} Update tests on beta
  • Currently blocked on update testing due to bug 767854 (server issue)

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-24
Windows Vista PASS - 2012-06-24
Windows 7 32-bit PASS - 2012-06-24
Windows 7 64-bit PASS - 2012-06-24
Mac OS X 10.6 PASS - 2012-06-24
Mac OS X 10.7 PASS - 2012-06-24
Ubuntu 11.10 32-bit PASS - 2012-06-24
Ubuntu 11.10 64-bit PASS - 2012-06-24
Test Failures (known bugs)
  • /restartTests/testDefaultBookmarks/test1.js: bug 767819
  • /testPrivateBrowsing/testAboutPrivateBrowsing.js: bug 767820
  • /testCookies/testRemoveAllCookies.js: bug 767821
  • /testSecurity/testGreyLarry.js: bug 767822

Manual Testing

Run manual smoke tests in Litmus.

  • Win 7 x86 - PASS - 2012-06-25 (Mihaela)
Focus Area Win7-32 WinXP Mac Linux
Plug-ins PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25
Top-sites PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25 PASS - 2012-06-25
Mac OSX 10.8 N/A N/A PASS - 2012-06-25 N/A

Exploratory testing etherpad: https://mozqa.etherpad.mozilla.org/14b9Exploratory

Plug-ins

  • Tests:
    • Flash 11.3
    • Java 6, 7
    • Silverlight
    • QuickTime
    • Try videos, games, chat, document viewing, or other rich interfaces, closing tabs/windows with loaded plug-in content, updating from older plugin versions (particularly with content already loaded and running), try reloading/new tab/new window with plugin content after update
  • Results:
    • Windows XP: PASS - 2012-06-25 (Simona)
    • Windows 7: PASS - 2012-06-25 (Paul)
    • Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
    • Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
  • Details & Bugs:

Top Site Compatibility

  • Tests:
    • YouTube: video playback, 3D, uploading large files (>100MB), commenting
    • Facebook: chat, status, pictures, video, games, timeline, groups
    • Twitter: tweet, follow, unfollow, media sharing, trends
    • Banking website authentication and transactions
    • E-Store authentication, shopping cart, and checkout
    • Webmail: Hotmail, Gmail, Yahoo
    • keep an eye out for crashes
  • Results:
    • Windows XP: PASS - 2012-06-25 (Simona)
    • Windows 7: PASS - 2012-06-25 (Paul)
    • Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
    • Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
  • Details & Bugs:

Mac OS X 10.8 Code-signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - 2012-06-25 - VladG
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - 2012-06-25 - VladG
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - 2012-06-25 - VladG
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - 2012-06-25 - VladG
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - 2012-06-25 - VladG
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - 2012-06-25 - VladG
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, Firefox should start up and browse without any issues
    • PASS - 2012-06-25 - VladG

Do the following testing to make sure the build works as expected...

  • Tests:
    • Try all the menu item functions
    • Gestures
    • Plug-in compatibility
      • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
    • Top-site compatibility
    • Add-on compatibility
    • Software update, especially with rich profiles and fallback updates (use betatest channel)
  • Results: PASS - 2012-06-25 - VladG
  • Details & Bugs:

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-24 PASS - 2012-06-25
Windows Vista PASS - 2012-06-24 PASS - 2012-06-25
Windows 7 32-bit PASS - 2012-06-24 PASS - 2012-06-25
Windows 7 64-bit PASS - 2012-06-24 PASS - 2012-06-25
Mac OS X 10.6 PASS - 2012-06-24 PASS - 2012-06-25
Mac OS 10.7 PASS - 2012-06-24 PASS - 2012-06-25
Ubuntu 11.10 32-bit PASS - 2012-06-24 PASS - 2012-06-25
Ubuntu 11.10 64-bit PASS - 2012-06-24 PASS - 2012-06-25
Issues

Bug Verifications

Add your name beside whatever you are testing so people don't duplicate your results.

  • bug 686335 - Bad plugin search order prevents upgrading Flash without an OS restart
  • bug 758363 - Implement NPN_ReloadPlugins for out-of-process Plugins
NOTE that testing any of these in a VM requires the VM to be run in Protected Mode

My advice would be to take a snapshot of the VM before starting and rolling back before each test they can be run from a clean snapshot.

NOTE that not all of these plugin versions are available for Mac and Linux.

Just test with whatever old versions you can find and update to the latest. Report the version numbers in your results.

Tests

Flash 10.2 (blocklisted)

Flash 10.3 (enabled & disabled)

  • Download and install Flash 10.3
  • Load some Flash content in tabs before updating
  • Pave-over install Flash 11.3.300.262
  • Reload tabs with Flash
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.2 (enabled & disabled)

  • Download and install Flash 11.2
  • Load some Flash content in tabs before updating
  • Pave-over install Flash 11.3.300.262
  • Reload tabs with Flash
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.3 (enabled & disabled)

  • Download and install Flash 11.3.300.257
  • Load some Flash content in tabs before updating
  • Pave-over Flash 11.3.300.262
  • Uninstall Flash and retest with Flash disabled before updating

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=true)

  • Download and install Flash 11.2
  • Start Firefox and open a page with flash content
  • Download but do not install Flash 11.3 Auto-update Installer
  • Open a command line to the download folder and run:
    • install_flash_player.exe -install -iv 9
  • Reload the page with Flash content once the installer finishes
  • Uninstall Flash
  • Repeat the test, loading Flash content in a new tab after update
  • Repeat the test, loading Flash content in a new window after update

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=false)

  • Download and install Flash 11.2
  • Start Firefox and open a page with flash content
  • Download but do not install Flash 11.3 Auto-update Installer
  • Open a command line to the download folder and run:
    • install_flash_player.exe -install -iv 9
  • Reload the page with Flash content once the installer finishes
  • Uninstall Flash
  • Repeat the test, loading Flash content in a new tab after update
  • Repeat the test, loading Flash content in a new window after update

Java 6u31 (enabled & disabled)

  • Download and install Java 6u31
  • Load some Java content in tabs before updating
  • Pave-over install Java 7u5
  • Reload tabs with Java content after updating
  • Uninstall Java and retest with Java disabled before updating

Java 7u2 (enabled & disabled)

  • Download and install Java 7u2
  • Load some Java content in tabs before updating
  • Pave-over install Java 7u5
  • Reload tabs with Java content after updating
  • Uninstall Java and retest with Java disabled before updating

Results

Expected

  • Plugin updates without any problems
  • Latest version of the plugin is listed in about:plugins
  • Content loads (or reloads) after update without crashing
  • NOTE: Multiple versions may be listed in about:plugins
    • OKAY as long as plugin content is using the latest version
    • Multiple entries can be cleared by restarting the OS

Windows 7 (ashughes)

  • PASS Flash 10.2 -> 11.3.300.262 (blocklisted) via pave-over
  • PASS Flash 10.3 -> 11.3.300.262 (enabled) via pave-over
  • PASS Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
  • PASS Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
  • PASS Flash 11.3 -> 11.3.300.262 (disabled) via auto-update
  • PASS Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
  • PASS Java 6u31 (disabled) -- requires Firefox restart
  • PASS Java 7u2 (enabled)

Windows XP (ashughes)

  • PASS Update Flash 10.2 -> 11.3.300.262 (disabled) via pave-over
  • PASS Update Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
  • PASS Update Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
  • PASS Update Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
  • PASS Update Java 6u31 -> 7u5 (disabled)
  • PASS Update Java 7u2 -> 7u5 (enabled)

Mac OSX 10.6 (ashughes)

  • PASS Update Flash 10.2 -> 11.3.300.257 (disabled) via pave-over
  • PASS Update Flash 11.2 -> 11.3.300.257 (enabled) via pave-over
  • PASS Update Flash 11.2 -> 11.3.300.257 (dom.ipc.plugins.enabled=false) via pave-over
  • result Update Java 6u* -> 7u5 (disabled) via Software Update
  • result Update Java 7u* -> 7u5 (enabled) via Software Update
  • Flash:
    • Flash installer forces Firefox to quit before updating
    • Disabling before update does not automatically enable after update

Ubuntu 12.04 (ashughes)

  • PASS Flash 11.2.205.233 -> 11.2.205.236 (enabled) via pave-over
  • PASS Flash 11.2.205.233 -> 11.2.205.236 (disabled) via pave-over
  • PASS Flash 11.2.205.233 -> 11.2.205.236 (dom.ipc.plugins.enabled=false) via pave-over
  • result Update Oracle Java 6u* -> 7u5 (disabled) via pave-over
  • result Update IcedTea Java Plugin (enabled) via Software Update
  • Flash Plugin:
    • Overwriting when Firefox is running will result in an expected recoverable plugin crash
    • Disabling a plugin before update does not automatically re-enable after pave-over update
  • Java: As of yet unable to find a reliable way to install/update an old version (Ubuntu only provides the latest)

Known Issues

  • Windows: Disabled Java requires restart of Firefox to re-enable after update
  • Side-by-Side Flash versions in about:config when paving over Flash 11.3.300.257 64-bit with 11.3.300.262 32-bit; resolved only by OS restart; does not seem to affect functionality/stability
    • NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
  • Side-by-Side Flash versions in about:config when simulating 11.2->11.3 auto-update with install_flash_player.exe -install -iv 9
    • NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
  • Hang/crash when watching Youtube after pave-over update (bug 686335#c63) -- not reproducible using auto-update

Issues Found

Full Query
ID Summary Priority Status
768002 crash in npjp2.dll@0x796b with Java SE 7 Update 5 -- RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
675645 Firefox freezes when using Pop out while in Fullscreen mode on Youtube P3 RESOLVED
711742 HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video -- RESOLVED
747858 A horizontal line is displayed on the Play button from HTML5 videos -- RESOLVED
754829 In Full Screen mode - YouTube videos player changes dimmension -- RESOLVED
760171 The inspector node menu feature allows the user to remove tabbed browsing P1 RESOLVED
760175 The :focus pseudoClass is not kept for the parent node when deleting a node -- RESOLVED
761990 On Windows the "Feedback" button's dropdown arrow overlaps the label -- RESOLVED
762011 [highlighter] Node operation bar displayed but not accessible for on mouse hover elements -- RESOLVED
762456 In full screen - Flash player behaves differently if opened from the button or context menu -- RESOLVED
762483 On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta -- RESOLVED

10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);


Beta 10

Builds FTP, HTTP
ID 20120628060610
Changeset e4445f905091
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [ON TRACK] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-06-28
Windows Vista PASS - 2012-06-28
Windows 7 32-bit PASS - 2012-06-28
Windows 7 64-bit PASS - 2012-06-28
Mac OS X 10.6 PASS - 2012-06-28
Mac OS X 10.7 PASS - 2012-06-28
Ubuntu 11.10 32-bit PASS - 2012-06-28
Ubuntu 11.10 64-bit PASS - 2012-06-28
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Simona PASS - 2012-06-28

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below. Results of manual testing (details in exploratory etherpad):

Platform Assignee Result: report
Windows 7 32-bit Simona PASS - 2012-06-29
Mac OS X 10.6.8 Vlad PASS - 2012-06-29
Ubuntu 11.10 64-bit Virgil PASS - 2012-06-29
Windows 7 64-bit Paul PASS - 2012-06-29

Exploratory Testing

[DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Opt-in activation for plugins Paul Silaghi RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-06-28 RESULT - 2012-06-28 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-06-28 RESULT - 2012-06-28 RESULT - 2012-06-28
Mac OS 10.8 Signing VladG n/a PASS - 2012-06-29 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-06-29
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-06-29
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-06-29
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-06-29
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-06-29
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-06-29
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-06-29

Spot check build after this test.

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
    • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
  • Top-site compatibility
  • Add-on compatibility
  • Software update, especially with rich profiles and fallback updates (use betatest channel)

Results: PASS - 2012-06-29 - VladG

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-06-28 RESULT - 2012-06-28
Windows Vista PASS - 2012-06-28 RESULT - 2012-06-28
Windows 7 32-bit PASS - 2012-06-28 RESULT - 2012-06-28
Windows 7 64-bit PASS - 2012-06-28 RESULT - 2012-06-28
Mac OS X 10.6 PASS - 2012-06-28 RESULT - 2012-06-28
Mac OS 10.7 PASS - 2012-06-28 RESULT - 2012-06-28
Ubuntu 11.10 32-bit PASS - 2012-06-28 RESULT - 2012-06-28
Ubuntu 11.10 64-bit PASS - 2012-06-28 RESULT - 2012-06-28
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 31 8 79%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Beta 11

Builds FTP, HTTP
ID 20120704090211
Changeset 48842c93fcb2
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [ON TRACK] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-07-04
Windows Vista PASS - 2012-07-04
Windows 7 32-bit PASS - 2012-07-04
Windows 7 64-bit PASS - 2012-07-04
Mac OS X 10.6 PASS - 2012-07-04
Mac OS X 10.7 PASS - 2012-07-04
Ubuntu 11.10 32-bit PASS - 2012-07-04
Ubuntu 11.10 64-bit PASS - 2012-07-04
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows XP Ioana PASS - 2012-07-05

Results in Litmus

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Opt-in activation for plugins Paul Silaghi RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-07-04 RESULT - 2012-07-04 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-07-04 RESULT - 2012-07-04 RESULT - 2012-07-04
Mac OS 10.8 Signing Required n/a RESULT - 2012-07-04 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-05
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-05
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-05
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-05
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-05
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-05
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-04-05

Spot check build after this test: PASS - VladG - 2012-04-05

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP RESULT - 2012-07-04 RESULT - 2012-07-04
Windows Vista RESULT - 2012-07-04 RESULT - 2012-07-04
Windows 7 32-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Windows 7 64-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Mac OS X 10.6 RESULT - 2012-07-04 RESULT - 2012-07-04
Mac OS 10.7 RESULT - 2012-07-04 RESULT - 2012-07-04
Ubuntu 11.10 32-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Ubuntu 11.10 64-bit RESULT - 2012-07-04 RESULT - 2012-07-04
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 31 8 79%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Known Issues

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Beta 12

Builds FTP, HTTP
ID 20120710123126
Changeset TBD
Checklist
  • [DONE] Functional automated tests, synced to Litmus
  • [DONE] Manual tests
  • [DONE] Bug fix verifications (target 50%)
  • [ON TRACK] Security fix verifications (target 25%)
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off from Bob Moss
  • [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-07-10
Windows Vista PASS - 2012-07-10
Windows 7 32-bit PASS - 2012-07-10
Windows 7 64-bit PASS - 2012-07-10
Mac OS X 10.6 PASS - 2012-07-10
Mac OS X 10.7 PASS - 2012-07-10
Ubuntu 11.10 32-bit PASS - 2012-07-10
Ubuntu 11.10 64-bit PASS - 2012-07-10
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows 7 32-bit Simona PASS - 2012-07-11

Results in Litmus

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature Owner Win32 Mac Linux
URL Autocomplete Vlad Ghetiu RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Opt-in activation for plugins Paul Silaghi RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Firefox migrators - Safari Migrator Paul Silaghi RESULT - 2012-07-10 RESULT - 2012-07-10 N/A
Pseudo-Class Lock Ioana Budnar RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Support for Mouse Capture / Mouse Lock Virgil Dicu RESULT - 2012-07-10 RESULT - 2012-07-10 RESULT - 2012-07-10
Mac OS 10.8 Signing Required n/a PASS - 2012-07-11 n/a
Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-11
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-11
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-11
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-11
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-11
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-11
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-11

Spot check build after this test - PASS - VladG - 2012-07-11

  • Try all the menu item functions
  • Gestures
  • Plug-in compatibility
    • When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-11 PASS - 2012-07-11
Windows Vista PASS - 2012-07-11 PASS - 2012-07-11
Windows 7 32-bit PASS - 2012-07-11 PASS - 2012-07-11
Windows 7 64-bit PASS - 2012-07-11 PASS - 2012-07-11
Mac OS X 10.6 PASS - 2012-07-11 PASS - 2012-07-11
Mac OS 10.7 PASS - 2012-07-11 PASS - 2012-07-11
Ubuntu 11.10 32-bit PASS - 2012-07-11 PASS - 2012-07-11
Ubuntu 11.10 64-bit PASS - 2012-07-11 PASS - 2012-07-11
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 36 3 92%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Full Query
ID Summary Priority Status
772786 Wrong favicons displayed in the Recently Bookmarked folder -- RESOLVED
772842 The newly opened tab is not always focused when taking a speed test from a youtube fullscreen video -- RESOLVED
773236 On Windows Vista Flash content is white after exiting Full Screen -- RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
675645 Firefox freezes when using Pop out while in Fullscreen mode on Youtube P3 RESOLVED
711545 Crash reports from last evening still not found -- VERIFIED
763237 [adbe 3221904] Firefox crash in F1398665248_____________________________ -- RESOLVED
770207 On Windows the "Feedback" button is not highlighted when hovering the mouse over it -- RESOLVED

4 Total; 0 Open (0%); 3 Resolved (75%); 1 Verified (25%);


Firefox 14.0.1 Release Candidates

Builds FTP, HTTP
ID 20120713134347
Changeset e5728a4e106c
Checklist
  • [DONE] Functional automated tests
  • [DONE] Manual tests
  • [DONE] Bug fix verifications
  • [ON TRACK] Security fix verifications
  • [DONE] Update tests on releasetest
  • [ON TRACK] Get final sign-off
  • [ON TRACK] Update tests on release

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform Result: report
Windows XP PASS - 2012-07-15
Windows Vista PASS - 2012-07-15
Windows 7 32-bit PASS - 2012-07-15
Windows 7 64-bit PASS - 2012-07-15
Mac OS X 10.6 PASS - 2012-07-15
Mac OS X 10.7 PASS - 2012-07-15
Ubuntu 11.10 32-bit PASS - 2012-07-15
Ubuntu 11.10 64-bit PASS - 2012-07-15
Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform Assignee Result
Windows 7 32-bit Mihaela PASS - 2012-07-16
Mac OS X 10.7.4 Ioana PASS - 2012-07-16
Ubuntu 12.04 32-bit Virgil PASS - 2012-07-16

Results in Litmus

Exploratory Testing

  1. [DONE] Exploratory testing on plugins, new features, web compatibility etc Etherpad

Feature Testing

By now we should have all features signed off.

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

  • Download the .dmg file and drag the Firefox.app file to a location on your computer
    • PASS - VladG - 2012-07-16
  • Open a terminal window and cd to the folder Firefox.app is located
    • PASS - VladG - 2012-07-16
  • Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
    • PASS - VladG - 2012-07-16
  • Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
    • PASS - VladG - 2012-07-16
  • Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
    • PASS - VladG - 2012-07-16
  • Click "open" and browse to any website, Firefox should start and you should be able to browse
    • PASS - VladG - 2012-07-16
  • Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
    • PASS - VladG - 2012-07-16

Spot check build after this test.

Try all the menu item functions - PASS - VladG - 2012-07-16

  • Gestures
  • Plug-in compatibility
  • Top-site compatibility
  • Add-on compatibility
  • Software update

Update Tests

The following are details of update verifications

Platform Releasetest: report Beta: report
Windows XP PASS - 2012-07-15 RESULT - 2012-07-15
Windows Vista PASS - 2012-07-15 RESULT - 2012-07-15
Windows 7 32-bit PASS - 2012-07-15 RESULT - 2012-07-15
Windows 7 64-bit PASS - 2012-07-15 RESULT - 2012-07-15
Mac OS X 10.6 PASS - 2012-07-15 RESULT - 2012-07-15
Mac OS 10.7 PASS - 2012-07-15 RESULT - 2012-07-15
Ubuntu 11.10 32-bit PASS - 2012-07-15 RESULT - 2012-07-15
Ubuntu 11.10 64-bit PASS - 2012-07-15 RESULT - 2012-07-15
Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

  • Any fix you very should have the status-firefox14 flag set to verified
  • If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query Triaged Remaining  % Complete
QA+: for testers to verify functional fixes 36 3 92%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

  • Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Full Query
ID Summary Priority Status
774212 Improve live bookmark loading indications P5 RESOLVED
774244 When HTML5 videos end - the suggestions are no longer displayed -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Known Issues

Full Query
ID Summary Priority Status
667399 RealPlayer G2 LiveConnect plugin crash -- RESOLVED
692446 Firefox prompts to install windows media player plugin even though it is already installed -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);