« Firefox 13 | Firefox 14 | Firefox 15 »

Firefox 14 Test Plan

Summary

Milestone	Firefox 14
Status	Currently in Beta -- Shipping July 17, 2012. Tentative.
Lead	Juan Becerra (irc: juanb)
2nd	Anthony Hughes (irc: ashughes)
Builds	[ftp] [http] [web]

The following is the test plan for Firefox 14 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 14. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

2012-03-13: [DONE] Nightly Merge
2012-04-24: [DONE] Aurora Merge
2012-06-05: [DONE] Pre-Beta feature sign-offs
2012-06-07: [DONE] Beta 6 (Starting at version 6 to sync with Firefox Mobile)
2012-06-14: [DONE] Beta 7
2012-06-21: [DONE] Beta 8
2012-06-28: [DONE] Beta 9
2012-07-05: [DONE] Beta 10
2012-07-11: [DONE] Beta 11
2012-07-13: [ON TRACK] Release Candidate
2012-07-17: {{|}} Final

Features

See the Feature Release Tracking wiki for a general overview

[DONE]

Cycle Collector Perf improvements (Virgil Dicu); landed in Firefox 13

[ON TRACK]

Windows8 Metro Firefox (Vlad Ghetiu)
Hang Detector and Reporter (Simona Badau)
URL Autocomplete (Vlad Ghetiu)
Firefox Migrators (Paul Silaghi)
Pseudo-Class Lock (Ioana Budnar)
Support for Mouse Capture / Mouse Lock (Virgil Dicu) - Signed off
Opt-in activation for plugins (Paul Silaghi); Initial minimal implementation landed for 14 (opt-in), further work in 15 and 16.

[AT RISK]

[MISSED]

Incremental garbage collection (Ioana Budnar); delayed to Firefox 15
Panel-based Download Manager (Simona Badau) opt-in; delayed to Firefox 15
Silent Update: Background updates (Vlad Ghetiu); delayed to Firefox 15
Windows Start-up Performance Improvements (Jason Smith); delayed to Firefox 15
Speedy Session Restore (Paul Silaghi); delayed to Firefox 15
Silent Update: Updated workflow for users with incompatible add-ons (Ioana Budnar); delayed to Firefox 17
Show PDF inline (Mihaela Velimiroviciu); delayed to Firefox 16
Web Apps Integration to Desktop (Jason Smith); delayed to Firefox 15

Nightly Merge

Functional Automation - result

Update Spotchecks - result

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do

Check the Endurance Charts
If you see an anomalous event (ie. spike or dip), get the report for the specific event
If the event happens twice in a row, file a bug under Mozmill Tests
Notify Juan Becerra of the issue
Record the event in this wiki below

Current Status (Mihaela Velimiroviciu) - Last Updated: 2012-06-25

bug 752887 - Increased memory usage for endurance tests on Linux Aurora branch
bug 768026 - Higher memory consumption on Win 6.0 on Beta branch since the merge

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do

Check the testrun reports
If you see any failing tests, check that there are not any bugs on file for it already
If not, file a bug under Mozmill Tests
Notify Juan Becerra of the issue
Record the event in this wiki below

Current Status (Vlad Maniac / Alex Lakatos) - Last Updated

No issues found

Unconfirmed Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs. The below charts provides a summary of the unconfirmed bugs currently in the Firefox product:

P1 Task - Get UNCONFIRMED Bugs Out of Firefox General and Untriaged If Possible

Check this <insert link> for UNCONFIRMED bugs found for this Firefox Release's Beta Timeframe
Read each bug and determine what product and component they belong under in bugzilla
Move the bug to the right product and component
CC any applicable team members that may be interested in the bug

P2 Task - Confirm UNCONFIRMED Bugs in this 6-Week Cycle

Check the <insert link> bugs
Test the bugs in the latest Firefox Nightly
If you can't reproduce the bug, ask for the reporter to provide more information
If you reproduce the bug, add your testing information and mark the bug NEW
Notify Juan Becerra of the NEW issue

P3 Task - Confirm UNCONFIRMED Bugs from Other Releases

Check the UNCONFIRMED bugs
Test the bugs in the latest Firefox Nightly
If you can't reproduce the bug, ask for the reporter to provide more information
If you reproduce the bug, add your testing information and mark the bug NEW
Notify Juan Becerra of the NEW issue

Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet

Aurora Merge

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-04-26
Windows Vista	PASS - 2012-04-26
Windows 7 32-bit	PASS - 2012-04-26
Windows 7 64-bit	PASS - 2012-04-26
Mac OS X 10.6	PASS - 2012-04-26
Mac OS X 10.7	PASS - 2012-04-26
Ubuntu 11.10 32-bit	PASS - 2012-04-26
Ubuntu 11.10 64-bit	PASS - 2012-04-26

Test Failures

/testPasswordManager/testPasswordNotificationBar.js (bug 749156)
/testPasswordManager/testPasswordSavedAndDeleted.js (bug 749206)
/testSecurity/testBlueLarry.js (bug 748686)
/testSecurity/testGreenLarry.js (bug 748686)
/testSecurity/testGreyLarry.js (bug 748686)

Updates

The following are details of update verifications for Firefox 12 merge to Aurora

Auroratest

[ON TRACK] [ashughes] Windows XP:
[ON TRACK] [ashughes] Windows 7:
[ON TRACK] [ashughes] Mac OS X 10.6:
[ON TRACK] [ashughes] Ubuntu 11.04:

Aurora

[ON TRACK] [ashughes] Windows XP:
[ON TRACK] [ashughes] Windows 7:
[ON TRACK] [ashughes] Mac OS X 10.6:
[ON TRACK] [ashughes] Ubuntu 11.04:

Issues Found

Manual Testing

Smoketests

Run the Aurora Merge Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows XP	Ioana	PASS - 2012-04-27
Windows Vista	Simona	PASS - 2012-04-27
Windows 7 32-bit	Paul	PASS - 2012-04-27
Windows 7 64-bit	Simona	PASS - 2012-04-27
Mac OS X 10.6.8	Paul	PASS - 2012-04-27
Mac OS X 10.7.3	Paul	PASS - 2012-04-27
Ubuntu 11.10 32-bit	Virgil	PASS - 2012-04-27
Ubuntu 11.10 64-bit	Mihaela	PASS - 2012-04-27

Results in Litmus

Feature Sign-offs

Run the Aurora 14.0 sign-offs tests in Litmus and report results below.

Feature	Owner	Win32	Mac	Linux
Panel-based Download Manager	Simona Badau	PASS - 2012-04-27	PASS - 2012-04-27	PASS - 2012-04-27
Windows 8 Metro Firefox	Vlad Ghetiu	N/A	N/A	N/A
Cycle Collector Perf improvements	Virgil Dicu	PASS - 2012-04-27	PASS - 2012-04-30	PASS - 2012-04-30
Incremental Garbage Collection	Ioana Budnar	PASS - 2012-04-27	PASS - 2012-04-27	PASS - 2012-04-27
Hang Detector and Reporter	Simona Badau	N/A	N/A	N/A
URL Autocomplete	Vlad Ghetiu	PASS - 2012-04-27	PASS - 2012-04-27	PASS - 2012-04-27
Opt-in activation for plugins	Paul Silaghi	PASS - 2012-04-27	PASS - 2012-04-30	PASS - 2012-04-30
Show PDF Inline	Mihaela Velimiroviciu	N/A	N/A	N/A
Firefox migrators	Paul Silaghi	PASS - 2012-04-30	PASS - 2012-04-30	N/A
Silent Update: Background updates	Vlad Ghetiu	N/A	N/A	N/A
Web Apps integration	Jason Smith	FAIL - 2012-05-01	FAIL - 2012-05-01	N/A - 2012-05-01
Pseudo-Class Lock	Ioana Budnar	PASS - 2012-04-30	PASS - 2012-04-30	PASS - 2012-04-30
Support for Mouse Capture / Mouse Lock	Virgil Dicu	FAIL - 2012-04-30	FAIL - 2012-04-30	FAIL - 2012-04-30

Results in Litmus

Notes:

Windows 8 Metro Firefox - still in the definition phase.
Hang Detector and Reporter - only available on the Nightly Profiling branch.
Silent Update: Background updates - didn't land in Aurora yet.
Show PDF Inline - feature is not available in latest Aurora builds (since Apr 26)

Issues Found

Full Query

ID	Summary	Priority	Status
749579	Visual glitch when playing chrome angrybirds in SD	--	RESOLVED
750245	Mouse lock pointer not working for quake demo	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
564934	Implement new Download Manager UI for browser	--	RESOLVED
726447	[meta] Theming and visual design	--	NEW
726453	[meta] Improve accessibility and keyboard interaction	--	NEW
728692	Javascript becomes slow after Incremental GC gets disabled	--	RESOLVED
735183	No Home Page Selection screen on first run import wizard	--	RESOLVED
735944	about:support says "incremental gc: 1" when javascript.options.mem.gc_incremental is disabled	--	VERIFIED
743102	Desktop Click to play blocks Silverlight - prompted to Install	--	VERIFIED
745187	Click-to-activate plugins which are removed immediately after adding don't trigger the plugin notification (Google Earth Plugin doesn't work with click-to-play enabled)	P1	VERIFIED
746658	Failed downloads can't be removed from panel	--	VERIFIED
746739	The new download list (when downloads button is clicked) looks bad	--	RESOLVED
746743	The new download list doesn't scroll automatically to the newly started download	--	RESOLVED
746751	Download panel icon doesn't appear	--	RESOLVED
746760	Tooltip covers part of the download list	--	NEW
746772	Downloads button disappears from toolbar when customizing	--	VERIFIED
746783	"Open Containing Folder" doesn't appear in context menu of downloads	P2	RESOLVED
747310	Custom toolbar disappears when Edit Bookmarks Panel or Downloads Panel pop up.	--	RESOLVED
747909	Youtube.com In Full Screen mode - copy actions are not allowed on HTML5 videos	--	RESOLVED
748381	Downloads indicator should persist in the toolbar (not auto-hide)	--	VERIFIED

18 Total; 3 Open (16.67%); 9 Resolved (50%); 6 Verified (33.33%);

Pre-beta features sign offs

Run the Firefox 14.0 pre-beta sign off tests in Litmus and report results below.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	PASS - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31
Opt-in activation for plugins	Paul Silaghi	PASS - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31
Firefox migrators - Safari Migrator	Paul Silaghi	PASS - 2012-05-31	PASS - 2012-05-31	N/A - 2012-05-31
Pseudo-Class Lock	Ioana Budnar	FAIL - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31
Support for Mouse Capture / Mouse Lock	Virgil Dicu	PASS - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31

Results in Litmus

Issues Found

Full Query

ID	Summary	Priority	Status
760171	The inspector node menu feature allows the user to remove tabbed browsing	P1	RESOLVED
760175	The :focus pseudoClass is not kept for the parent node when deleting a node	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Known Issues

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Beta 6


Builds	FTP, HTTP
ID	20120605113340
Changeset	89ec8943347a

Checklist

[DONE] Functional automated tests, synced to Litmus
[DONE] Manual tests
[ON TRACK] Bug fix verifications (target 50%)
[ON TRACK] Security fix verifications (target 25%)
[DONE] Update tests on releasetest
[DONE] Get final sign-off from Bob Moss
[DONE] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-06-06
Windows Vista	PASS - 2012-06-06
Windows 7 32-bit	PASS - 2012-06-06
Windows 7 64-bit	PASS - 2012-06-06
Mac OS X 10.6	PASS - 2012-06-06
Mac OS X 10.7	PASS - 2012-06-06
Ubuntu 11.10 32-bit	PASS - 2012-06-06
Ubuntu 11.10 64-bit	PASS - 2012-06-06

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Working on cloning the branch as of 06-04 @11:50pm Pacific

Platform	Assignee	Result
Windows XP	Simona	PASS - 2012-06-06
Windows Vista	tester	RESULT - 2012-06-06
Windows 7 32-bit	tester	RESULT - 2012-06-06
Windows 7 64-bit	tester	RESULT - 2012-06-06
Mac OS X 10.6.8	tester	RESULT - 2012-06-06
Mac OS X 10.7.4	tester	RESULT - 2012-06-06
Ubuntu 12.04 32-bit	tester	RESULT - 2012-06-06
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-06
Windows 7 32-bit (es-AR)	Gabriela	PASS - 2012-06-06

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform	Assignee	Result: report
Windows 7 32-bit	tester	RESULT - 2012-06-06
Mac OS X 10.6.8	tester	RESULT - 2012-06-06
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-06
Windows XP	tester	RESULT - 2012-06-06
Windows 7 32-bit (es-AR)	Gabriela	PASS - 2012-06-07

Exploratory Testing

[DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
[DONE] Exploratory testing on the new features Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06
Opt-in activation for plugins	Paul Silaghi	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06
Firefox migrators - Safari Migrator	Paul Silaghi	PASS - 2012-06-06	PASS - 2012-06-06	N/A
Pseudo-Class Lock	Ioana Budnar	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06
Support for Mouse Capture / Mouse Lock	Virgil Dicu	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
Open a terminal window and cd to the folder Firefox.app is located
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
Click "open" and browse to any website, Firefox should start and you should be able to browse
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues

Spot check build after this test.

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	PASS - 2012-06-06	RESULT - 2012-06-06
Windows Vista	PASS - 2012-06-06	RESULT - 2012-06-06
Windows 7 32-bit	PASS - 2012-06-06	RESULT - 2012-06-06
Windows 7 64-bit	PASS - 2012-06-06	RESULT - 2012-06-06
Mac OS X 10.6	PASS - 2012-06-06	RESULT - 2012-06-06
Mac OS 10.7	PASS - 2012-06-06	RESULT - 2012-06-06
Ubuntu 11.10 32-bit	PASS - 2012-06-06	RESULT - 2012-06-06
Ubuntu 11.10 64-bit	PASS - 2012-06-06	RESULT - 2012-06-06

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Any fix you very should have the status-firefox14 flag set to verified
If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	14	25	36%
QA+ SG: for the Security team to verify security fixes	7	23	23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs	297	0	100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Need monitoring of crash stats for bug 760190
Need steps to reproduce bug 709860
bug 755728 only happens on mobile
bug 748047 needs verification (rest profile)
bug 748569 needs verification (migration code)

Issues Found

Full Query

ID	Summary	Priority	Status
761990	On Windows the "Feedback" button's dropdown arrow overlaps the label	--	RESOLVED
762011	[highlighter] Node operation bar displayed but not accessible for on mouse hover elements	--	RESOLVED
762456	In full screen - Flash player behaves differently if opened from the button or context menu	--	RESOLVED
762483	On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta	--	RESOLVED

4 Total; 0 Open (0%); 4 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
675645	Firefox freezes when using Pop out while in Fullscreen mode on Youtube	P3	RESOLVED
711742	HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video	--	RESOLVED
747858	A horizontal line is displayed on the Play button from HTML5 videos	--	RESOLVED
754829	In Full Screen mode - YouTube videos player changes dimmension	--	RESOLVED
760171	The inspector node menu feature allows the user to remove tabbed browsing	P1	RESOLVED
760175	The :focus pseudoClass is not kept for the parent node when deleting a node	--	RESOLVED

6 Total; 0 Open (0%); 6 Resolved (100%); 0 Verified (0%);

Beta 6 Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 7


Builds	FTP, HTTP
ID	TBD
Changeset	89ec8943347a

Checklist

[DONE] Functional automated tests, synced to Litmus
[DONE] Manual tests
[DONE] Bug fix verifications (target 50%)
[ON TRACK] Security fix verifications (target 25%)
[DONE] Update tests on releasetest
[ON TRACK] Get final sign-off from Bob Moss
[ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-06-12
Windows Vista	PASS - 2012-06-12
Windows 7 32-bit	PASS - 2012-06-12
Windows 7 64-bit	PASS - 2012-06-12
Mac OS X 10.6	PASS - 2012-06-12
Mac OS X 10.7	PASS - 2012-06-12
Ubuntu 11.10 32-bit	PASS - 2012-06-12
Ubuntu 11.10 64-bit	PASS - 2012-06-12

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows XP	tester	RESULT - 2012-06-13
Windows Vista	tester	RESULT - 2012-06-13
Windows 7 32-bit	Simona	In progress - 2012-06-13
Windows 7 64-bit	tester	RESULT - 2012-06-13
Mac OS X 10.6.8	VladG	PASS - 2012-06-13
Mac OS X 10.7.4	tester	RESULT - 2012-06-13
Ubuntu 12.04 32-bit	tester	RESULT - 2012-06-13
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-13
Windows 7 32-bit (es-AR)	Gabriela	RESULT - 2012-06-13

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform	Assignee	Result: report
Windows 7 32-bit	tester	RESULT - 2012-06-12
Mac OS X 10.6.8	tester	RESULT - 2012-06-12
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-12
Windows XP	tester	RESULT - 2012-06-12
Windows 7 32-bit (es-AR)	Gabriela	PASS - 2012-06-07

Exploratory Testing

[DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad
[DONE] Exploratory testing on the new features Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	RESULT - 2012-06-12	RESULT - 2012-06-12	RESULT - 2012-06-12
Opt-in activation for plugins	Paul Silaghi	RESULT - 2012-06-12	RESULT - 2012-06-12	RESULT - 2012-06-12
Firefox migrators - Safari Migrator	Paul Silaghi	RESULT - 2012-06-12	RESULT - 2012-06-12	N/A
Pseudo-Class Lock	Ioana Budnar	RESULT - 2012-06-12	RESULT - 2012-06-12	RESULT - 2012-06-12
Support for Mouse Capture / Mouse Lock	Virgil Dicu	RESULT - 2012-06-12	RESULT - 2012-06-12	RESULT - 2012-06-12
Mac OS 10.8 Signing	Required	n/a	RESULT - 2012-06-12	n/a

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG
Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - the output is containing "com.apple.FinderInfo" and "com.apple.quarantine"
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG
Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG

Spot check build after this test.

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	PASS - 2012-06-12	RESULT - 2012-06-12
Windows Vista	PASS - 2012-06-12	RESULT - 2012-06-12
Windows 7 32-bit	PASS - 2012-06-12	RESULT - 2012-06-12
Windows 7 64-bit	PASS - 2012-06-12	RESULT - 2012-06-12
Mac OS X 10.6	PASS - 2012-06-12	RESULT - 2012-06-12
Mac OS 10.7	PASS - 2012-06-12	RESULT - 2012-06-12
Ubuntu 11.10 32-bit	PASS - 2012-06-12	RESULT - 2012-06-12
Ubuntu 11.10 64-bit	PASS - 2012-06-12	RESULT - 2012-06-12

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Any fix you very should have the status-firefox14 flag set to verified
If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	20	19	51%
QA+ SG: for the Security team to verify security fixes	7	23	23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs	297	0	100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Need monitoring of crash stats for bug 760190
bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
bug 761990 Feedback button regression - investigated and updated (Simona).
bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
bug 748569 needs verification (migration code)
Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
bug 755728 low priority investigation - investigated and updated (Paul).

Issues Found

Full Query

ID	Summary	Priority	Status
764366	When logging into a site with multiple accounts and only saving the passwords for some of them, not all emails/usernames are remembered	--	RESOLVED
764385	[10.7][10.8]Silverlight keeps on crashing on Firefox 14 beta	--	RESOLVED
764388	"http://www." is appended to the string entered in the location bar, when it contains at least one dot , even if the string after the dot is obviously not a domain name	--	RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
587874	Undo Close Tab should only open tabs from the current group	P1	RESOLVED
675645	Firefox freezes when using Pop out while in Fullscreen mode on Youtube	P3	RESOLVED
711742	HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video	--	RESOLVED
747858	A horizontal line is displayed on the Play button from HTML5 videos	--	RESOLVED
754829	In Full Screen mode - YouTube videos player changes dimmension	--	RESOLVED
760171	The inspector node menu feature allows the user to remove tabbed browsing	P1	RESOLVED
760175	The :focus pseudoClass is not kept for the parent node when deleting a node	--	RESOLVED
761990	On Windows the "Feedback" button's dropdown arrow overlaps the label	--	RESOLVED
762011	[highlighter] Node operation bar displayed but not accessible for on mouse hover elements	--	RESOLVED
762456	In full screen - Flash player behaves differently if opened from the button or context menu	--	RESOLVED
762483	On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta	--	RESOLVED
764062	Autofilled suggestions in address bar do not include port numbers	P3	RESOLVED

12 Total; 0 Open (0%); 12 Resolved (100%); 0 Verified (0%);

Beta 7 Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 8


Builds	FTP, HTTP
ID	20120619191901
Changeset	Changeset

Checklist

[DONE] Functional automated tests, synced to Litmus
[DONE] Manual tests
[ON TRACK] Bug fix verifications (target 50%)
[ON TRACK] Security fix verifications (target 25%)
[DONE] Update tests on releasetest
[ON TRACK] Get final sign-off from Bob Moss
[ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-06-20
Windows Vista	PASS - 2012-06-20
Windows 7 32-bit	PASS - 2012-06-20
Windows 7 64-bit	PASS - 2012-06-20
Mac OS X 10.6	PASS - 2012-06-20
Mac OS X 10.7	PASS - 2012-06-20
Ubuntu 11.10 32-bit	PASS - 2012-06-20
Ubuntu 11.10 64-bit	PASS - 2012-06-20

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows XP	Simona	PASS - 2012-06-19
Windows Vista	tester	RESULT - 2012-06-19
Windows 7 32-bit	tester	RESULT - 2012-06-19
Windows 7 64-bit	tester	RESULT - 2012-06-19
Mac OS X 10.6.8	tester	RESULT - 2012-06-19
Mac OS X 10.7.4	tester	RESULT - 2012-06-19
Ubuntu 12.04 32-bit	tester	RESULT - 2012-06-19
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-19
Windows 8 32-bit build (en-US)	juanb	PASS - 2012-06-20

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform	Assignee	Result: report
Windows 7 32-bit	tester	RESULT - 2012-06-19
Mac OS X 10.6.8	tester	RESULT - 2012-06-19
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-19
Windows XP	tester	RESULT - 2012-06-19

Exploratory Testing

[DONE] Exploratory testing on plugins, addons, bookmarks, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	RESULT - 2012-06-19	RESULT - 2012-06-19	RESULT - 2012-06-19
Opt-in activation for plugins	Paul Silaghi	RESULT - 2012-06-19	RESULT - 2012-06-19	RESULT - 2012-06-19
Firefox migrators - Safari Migrator	Paul Silaghi	RESULT - 2012-06-19	RESULT - 2012-06-19	N/A
Pseudo-Class Lock	Ioana Budnar	RESULT - 2012-06-19	RESULT - 2012-06-19	RESULT - 2012-06-19
Support for Mouse Capture / Mouse Lock	Virgil Dicu	RESULT - 2012-06-19	RESULT - 2012-06-19	RESULT - 2012-06-19
Mac OS 10.8 Signing	Required	n/a	RESULT - 2012-06-19	n/a

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS (Ioana)
Open a terminal window and cd to the folder Firefox.app is located
- PASS (Ioana)
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS (Ioana)
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS (Ioana)
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS (Ioana)
Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS (Ioana)
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS (Ioana)

Spot check build after this test. Results

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	RESULT - 2012-06-19	RESULT - 2012-06-19
Windows Vista	RESULT - 2012-06-19	RESULT - 2012-06-19
Windows 7 32-bit	RESULT - 2012-06-19	RESULT - 2012-06-19
Windows 7 64-bit	RESULT - 2012-06-19	RESULT - 2012-06-19
Mac OS X 10.6	RESULT - 2012-06-19	RESULT - 2012-06-19
Mac OS 10.7	RESULT - 2012-06-19	RESULT - 2012-06-19
Ubuntu 11.10 32-bit	RESULT - 2012-06-19	RESULT - 2012-06-19
Ubuntu 11.10 64-bit	RESULT - 2012-06-19	RESULT - 2012-06-19

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Any fix you very should have the status-firefox14 flag set to verified
If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	7	32	89%
SG with testcase: for the Security team to verify security fixes	7	23	23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs	297	0	100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Need monitoring of crash stats for bug 760190
bug 746036#c13 needs exploratory testing with Yahoo Toolbar / Trusteer Rapport - investigated and updated (Paul).
bug 761990 Feedback button regression - investigated and updated (Simona).
bug 748047 needs verification (reset profile) - verified as fixed (Ioana).
bug 748569 needs verification (migration code)
Need steps to reproduce bug 709860 - investigated and commented, no STR yet (Virgil)
bug 755728 low priority investigation - investigated and updated (Paul).
bug 764157 regression range wanted - found regression range/updated (Virgil)
bug 754000 performance regression - investigated and updated (Paul).

Issues Found

Full Query

ID	Summary	Priority	Status
766548	[adbe 3226428] Taking a speed test on YouTube hangs Firefox	--	RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
675645	Firefox freezes when using Pop out while in Fullscreen mode on Youtube	P3	RESOLVED
711742	HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video	--	RESOLVED
747858	A horizontal line is displayed on the Play button from HTML5 videos	--	RESOLVED
754829	In Full Screen mode - YouTube videos player changes dimmension	--	RESOLVED
760171	The inspector node menu feature allows the user to remove tabbed browsing	P1	RESOLVED
760175	The :focus pseudoClass is not kept for the parent node when deleting a node	--	RESOLVED
761990	On Windows the "Feedback" button's dropdown arrow overlaps the label	--	RESOLVED
762011	[highlighter] Node operation bar displayed but not accessible for on mouse hover elements	--	RESOLVED
762456	In full screen - Flash player behaves differently if opened from the button or context menu	--	RESOLVED
762483	On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta	--	RESOLVED

10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);

Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.

Beta 9

NOTE this is a chemspill Beta to mitigate requirement to restart OS on Flash update


Builds	FTP, HTTP
ID	20120624012213
Changeset	d050090e578c

Checklist

[DONE] Functional automated tests
[ON TRACK] Manual tests
[ON TRACK] Bug fix verifications
[DONE] Update tests on releasetest
{{ |}} Peer sign-off
{{ |}} Update tests on beta
Currently blocked on update testing due to bug 767854 (server issue)

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-06-24
Windows Vista	PASS - 2012-06-24
Windows 7 32-bit	PASS - 2012-06-24
Windows 7 64-bit	PASS - 2012-06-24
Mac OS X 10.6	PASS - 2012-06-24
Mac OS X 10.7	PASS - 2012-06-24
Ubuntu 11.10 32-bit	PASS - 2012-06-24
Ubuntu 11.10 64-bit	PASS - 2012-06-24

Test Failures (known bugs)

/restartTests/testDefaultBookmarks/test1.js: bug 767819
/testPrivateBrowsing/testAboutPrivateBrowsing.js: bug 767820
/testCookies/testRemoveAllCookies.js: bug 767821
/testSecurity/testGreyLarry.js: bug 767822

Manual Testing

Run manual smoke tests in Litmus.

Win 7 x86 - PASS - 2012-06-25 (Mihaela)

Focus Area	Win7-32	WinXP	Mac	Linux
Plug-ins	PASS - 2012-06-25	PASS - 2012-06-25	PASS - 2012-06-25	PASS - 2012-06-25
Top-sites	PASS - 2012-06-25	PASS - 2012-06-25	PASS - 2012-06-25	PASS - 2012-06-25
Mac OSX 10.8	N/A	N/A	PASS - 2012-06-25	N/A

Exploratory testing etherpad: https://mozqa.etherpad.mozilla.org/14b9Exploratory

Plug-ins

Tests:
- Flash 11.3
- Java 6, 7
- Silverlight
- QuickTime
- Try videos, games, chat, document viewing, or other rich interfaces, closing tabs/windows with loaded plug-in content, updating from older plugin versions (particularly with content already loaded and running), try reloading/new tab/new window with plugin content after update
Results:
- Windows XP: PASS - 2012-06-25 (Simona)
- Windows 7: PASS - 2012-06-25 (Paul)
- Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
- Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
Details & Bugs:

Top Site Compatibility

Tests:
- YouTube: video playback, 3D, uploading large files (>100MB), commenting
- Facebook: chat, status, pictures, video, games, timeline, groups
- Twitter: tweet, follow, unfollow, media sharing, trends
- Banking website authentication and transactions
- E-Store authentication, shopping cart, and checkout
- Webmail: Hotmail, Gmail, Yahoo
- keep an eye out for crashes
Results:
- Windows XP: PASS - 2012-06-25 (Simona)
- Windows 7: PASS - 2012-06-25 (Paul)
- Ubuntu 12.04: PASS - 2012-06-25 (Virgil)
- Mac OSX 10.6.8: PASS - 2012-06-25 (Vladg)
Details & Bugs:

Mac OS X 10.8 Code-signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - 2012-06-25 - VladG
Open a terminal window and cd to the folder Firefox.app is located
- PASS - 2012-06-25 - VladG
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - 2012-06-25 - VladG
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - 2012-06-25 - VladG
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - 2012-06-25 - VladG
Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - 2012-06-25 - VladG
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, Firefox should start up and browse without any issues
- PASS - 2012-06-25 - VladG

Do the following testing to make sure the build works as expected...

Tests:
- Try all the menu item functions
- Gestures
- Plug-in compatibility
  - When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
- Top-site compatibility
- Add-on compatibility
- Software update, especially with rich profiles and fallback updates (use betatest channel)
Results: PASS - 2012-06-25 - VladG
Details & Bugs:

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	PASS - 2012-06-24	PASS - 2012-06-25
Windows Vista	PASS - 2012-06-24	PASS - 2012-06-25
Windows 7 32-bit	PASS - 2012-06-24	PASS - 2012-06-25
Windows 7 64-bit	PASS - 2012-06-24	PASS - 2012-06-25
Mac OS X 10.6	PASS - 2012-06-24	PASS - 2012-06-25
Mac OS 10.7	PASS - 2012-06-24	PASS - 2012-06-25
Ubuntu 11.10 32-bit	PASS - 2012-06-24	PASS - 2012-06-25
Ubuntu 11.10 64-bit	PASS - 2012-06-24	PASS - 2012-06-25

Issues

Bug Verifications

Add your name beside whatever you are testing so people don't duplicate your results.

bug 686335 - Bad plugin search order prevents upgrading Flash without an OS restart
bug 758363 - Implement NPN_ReloadPlugins for out-of-process Plugins

NOTE that testing any of these in a VM requires the VM to be run in Protected Mode

My advice would be to take a snapshot of the VM before starting and rolling back before each test they can be run from a clean snapshot.

NOTE that not all of these plugin versions are available for Mac and Linux.

Just test with whatever old versions you can find and update to the latest. Report the version numbers in your results.

Tests

Flash 10.2 (blocklisted)

Download and install Flash 10.2
Blocklist Flash by forcing a ping
Load some Flash content in tabs before updating
Pave-over install Flash 11.3.300.262
Reload tabs with Flash

Flash 10.3 (enabled & disabled)

Download and install Flash 10.3
Load some Flash content in tabs before updating
Pave-over install Flash 11.3.300.262
Reload tabs with Flash
Uninstall Flash and retest with Flash disabled before updating

Flash 11.2 (enabled & disabled)

Download and install Flash 11.2
Load some Flash content in tabs before updating
Pave-over install Flash 11.3.300.262
Reload tabs with Flash
Uninstall Flash and retest with Flash disabled before updating

Flash 11.3 (enabled & disabled)

Download and install Flash 11.3.300.257
Load some Flash content in tabs before updating
Pave-over Flash 11.3.300.262
Uninstall Flash and retest with Flash disabled before updating

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=true)

Download and install Flash 11.2
Start Firefox and open a page with flash content
Download but do not install Flash 11.3 Auto-update Installer
Open a command line to the download folder and run:
- install_flash_player.exe -install -iv 9
Reload the page with Flash content once the installer finishes
Uninstall Flash
Repeat the test, loading Flash content in a new tab after update
Repeat the test, loading Flash content in a new window after update

Flash 11.3 Automatic Update (dom.ipc.plugins.enabled=false)

Download and install Flash 11.2
Start Firefox and open a page with flash content
Download but do not install Flash 11.3 Auto-update Installer
Open a command line to the download folder and run:
- install_flash_player.exe -install -iv 9
Reload the page with Flash content once the installer finishes
Uninstall Flash
Repeat the test, loading Flash content in a new tab after update
Repeat the test, loading Flash content in a new window after update

Java 6u31 (enabled & disabled)

Download and install Java 6u31
Load some Java content in tabs before updating
Pave-over install Java 7u5
Reload tabs with Java content after updating
Uninstall Java and retest with Java disabled before updating

Java 7u2 (enabled & disabled)

Download and install Java 7u2
Load some Java content in tabs before updating
Pave-over install Java 7u5
Reload tabs with Java content after updating
Uninstall Java and retest with Java disabled before updating

Results

Expected

Plugin updates without any problems
Latest version of the plugin is listed in about:plugins
Content loads (or reloads) after update without crashing
NOTE: Multiple versions may be listed in about:plugins
- OKAY as long as plugin content is using the latest version
- Multiple entries can be cleared by restarting the OS

Windows 7 (ashughes)

PASS Flash 10.2 -> 11.3.300.262 (blocklisted) via pave-over
PASS Flash 10.3 -> 11.3.300.262 (enabled) via pave-over
PASS Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
PASS Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
PASS Flash 11.3 -> 11.3.300.262 (disabled) via auto-update
PASS Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
PASS Java 6u31 (disabled) -- requires Firefox restart
PASS Java 7u2 (enabled)

Windows XP (ashughes)

PASS Update Flash 10.2 -> 11.3.300.262 (disabled) via pave-over
PASS Update Flash 11.2 -> 11.3.300.262 (enabled) via pave-over
PASS Update Flash 11.3 -> 11.3.300.262 (enabled) via pave-over
PASS Update Flash 11.3 -> 11.3.300.262 (dom.ipc.plugins.enabled=false) via auto-update
PASS Update Java 6u31 -> 7u5 (disabled)
PASS Update Java 7u2 -> 7u5 (enabled)

Mac OSX 10.6 (ashughes)

PASS Update Flash 10.2 -> 11.3.300.257 (disabled) via pave-over
PASS Update Flash 11.2 -> 11.3.300.257 (enabled) via pave-over
PASS Update Flash 11.2 -> 11.3.300.257 (dom.ipc.plugins.enabled=false) via pave-over
result Update Java 6u* -> 7u5 (disabled) via Software Update
result Update Java 7u* -> 7u5 (enabled) via Software Update
Flash:
- Flash installer forces Firefox to quit before updating
- Disabling before update does not automatically enable after update

Ubuntu 12.04 (ashughes)

PASS Flash 11.2.205.233 -> 11.2.205.236 (enabled) via pave-over
PASS Flash 11.2.205.233 -> 11.2.205.236 (disabled) via pave-over
PASS Flash 11.2.205.233 -> 11.2.205.236 (dom.ipc.plugins.enabled=false) via pave-over
result Update Oracle Java 6u* -> 7u5 (disabled) via pave-over
result Update IcedTea Java Plugin (enabled) via Software Update
Flash Plugin:
- Overwriting when Firefox is running will result in an expected recoverable plugin crash
- Disabling a plugin before update does not automatically re-enable after pave-over update
Java: As of yet unable to find a reliable way to install/update an old version (Ubuntu only provides the latest)

Known Issues

Windows: Disabled Java requires restart of Firefox to re-enable after update
Side-by-Side Flash versions in about:config when paving over Flash 11.3.300.257 64-bit with 11.3.300.262 32-bit; resolved only by OS restart; does not seem to affect functionality/stability
- NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
Side-by-Side Flash versions in about:config when simulating 11.2->11.3 auto-update with install_flash_player.exe -install -iv 9
- NOTE: This is expected as long as newly loaded or reloaded Flash content uses the latest version
Hang/crash when watching Youtube after pave-over update (bug 686335#c63) -- not reproducible using auto-update

Issues Found

Full Query

ID	Summary	Priority	Status
768002	crash in npjp2.dll@0x796b with Java SE 7 Update 5	--	RESOLVED

1 Total; 0 Open (0%); 1 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
675645	Firefox freezes when using Pop out while in Fullscreen mode on Youtube	P3	RESOLVED
711742	HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video	--	RESOLVED
747858	A horizontal line is displayed on the Play button from HTML5 videos	--	RESOLVED
754829	In Full Screen mode - YouTube videos player changes dimmension	--	RESOLVED
760171	The inspector node menu feature allows the user to remove tabbed browsing	P1	RESOLVED
760175	The :focus pseudoClass is not kept for the parent node when deleting a node	--	RESOLVED
761990	On Windows the "Feedback" button's dropdown arrow overlaps the label	--	RESOLVED
762011	[highlighter] Node operation bar displayed but not accessible for on mouse hover elements	--	RESOLVED
762456	In full screen - Flash player behaves differently if opened from the button or context menu	--	RESOLVED
762483	On Windows the context menu is not displayed in Full Screen using Flash 11.3 beta	--	RESOLVED

10 Total; 0 Open (0%); 10 Resolved (100%); 0 Verified (0%);

Beta 10


Builds	FTP, HTTP
ID	20120628060610
Changeset	e4445f905091

Checklist

[DONE] Functional automated tests, synced to Litmus
[DONE] Manual tests
[ON TRACK] Bug fix verifications (target 50%)
[ON TRACK] Security fix verifications (target 25%)
[DONE] Update tests on releasetest
[ON TRACK] Get final sign-off from Bob Moss
[ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-06-28
Windows Vista	PASS - 2012-06-28
Windows 7 32-bit	PASS - 2012-06-28
Windows 7 64-bit	PASS - 2012-06-28
Mac OS X 10.6	PASS - 2012-06-28
Mac OS X 10.7	PASS - 2012-06-28
Ubuntu 11.10 32-bit	PASS - 2012-06-28
Ubuntu 11.10 64-bit	PASS - 2012-06-28

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows XP	Simona	PASS - 2012-06-28

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below. Results of manual testing (details in exploratory etherpad):

Platform	Assignee	Result: report
Windows 7 32-bit	Simona	PASS - 2012-06-29
Mac OS X 10.6.8	Vlad	PASS - 2012-06-29
Ubuntu 11.10 64-bit	Virgil	PASS - 2012-06-29
Windows 7 64-bit	Paul	PASS - 2012-06-29

Exploratory Testing

[DONE] Exploratory testing on plugins, addons, bookmarks etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	RESULT - 2012-06-28	RESULT - 2012-06-28	RESULT - 2012-06-28
Opt-in activation for plugins	Paul Silaghi	RESULT - 2012-06-28	RESULT - 2012-06-28	RESULT - 2012-06-28
Firefox migrators - Safari Migrator	Paul Silaghi	RESULT - 2012-06-28	RESULT - 2012-06-28	N/A
Pseudo-Class Lock	Ioana Budnar	RESULT - 2012-06-28	RESULT - 2012-06-28	RESULT - 2012-06-28
Support for Mouse Capture / Mouse Lock	Virgil Dicu	RESULT - 2012-06-28	RESULT - 2012-06-28	RESULT - 2012-06-28
Mac OS 10.8 Signing	VladG	n/a	PASS - 2012-06-29	n/a

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-06-29
Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-06-29
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-06-29
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-06-29
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-06-29
Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-06-29
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-06-29

Spot check build after this test.

Try all the menu item functions
Gestures
Plug-in compatibility
- When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
Top-site compatibility
Add-on compatibility
Software update, especially with rich profiles and fallback updates (use betatest channel)

Results: PASS - 2012-06-29 - VladG

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	PASS - 2012-06-28	RESULT - 2012-06-28
Windows Vista	PASS - 2012-06-28	RESULT - 2012-06-28
Windows 7 32-bit	PASS - 2012-06-28	RESULT - 2012-06-28
Windows 7 64-bit	PASS - 2012-06-28	RESULT - 2012-06-28
Mac OS X 10.6	PASS - 2012-06-28	RESULT - 2012-06-28
Mac OS 10.7	PASS - 2012-06-28	RESULT - 2012-06-28
Ubuntu 11.10 32-bit	PASS - 2012-06-28	RESULT - 2012-06-28
Ubuntu 11.10 64-bit	PASS - 2012-06-28	RESULT - 2012-06-28

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Any fix you very should have the status-firefox14 flag set to verified
If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	31	8	79%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Beta 11


Builds	FTP, HTTP
ID	20120704090211
Changeset	48842c93fcb2

Checklist

[DONE] Functional automated tests, synced to Litmus
[DONE] Manual tests
[DONE] Bug fix verifications (target 50%)
[ON TRACK] Security fix verifications (target 25%)
[ON TRACK] Update tests on releasetest
[ON TRACK] Get final sign-off from Bob Moss
[ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-07-04
Windows Vista	PASS - 2012-07-04
Windows 7 32-bit	PASS - 2012-07-04
Windows 7 64-bit	PASS - 2012-07-04
Mac OS X 10.6	PASS - 2012-07-04
Mac OS X 10.7	PASS - 2012-07-04
Ubuntu 11.10 32-bit	PASS - 2012-07-04
Ubuntu 11.10 64-bit	PASS - 2012-07-04

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows XP	Ioana	PASS - 2012-07-05

Results in Litmus

Exploratory Testing

[DONE] Exploratory testing on plugins, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	RESULT - 2012-07-04	RESULT - 2012-07-04	RESULT - 2012-07-04
Opt-in activation for plugins	Paul Silaghi	RESULT - 2012-07-04	RESULT - 2012-07-04	RESULT - 2012-07-04
Firefox migrators - Safari Migrator	Paul Silaghi	RESULT - 2012-07-04	RESULT - 2012-07-04	N/A
Pseudo-Class Lock	Ioana Budnar	RESULT - 2012-07-04	RESULT - 2012-07-04	RESULT - 2012-07-04
Support for Mouse Capture / Mouse Lock	Virgil Dicu	RESULT - 2012-07-04	RESULT - 2012-07-04	RESULT - 2012-07-04
Mac OS 10.8 Signing	Required	n/a	RESULT - 2012-07-04	n/a

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-07-05
Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-07-05
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-07-05
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-07-05
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-07-05
Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-07-05
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-04-05

Spot check build after this test: PASS - VladG - 2012-04-05

Try all the menu item functions
Gestures
Plug-in compatibility
Top-site compatibility
Add-on compatibility

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	RESULT - 2012-07-04	RESULT - 2012-07-04
Windows Vista	RESULT - 2012-07-04	RESULT - 2012-07-04
Windows 7 32-bit	RESULT - 2012-07-04	RESULT - 2012-07-04
Windows 7 64-bit	RESULT - 2012-07-04	RESULT - 2012-07-04
Mac OS X 10.6	RESULT - 2012-07-04	RESULT - 2012-07-04
Mac OS 10.7	RESULT - 2012-07-04	RESULT - 2012-07-04
Ubuntu 11.10 32-bit	RESULT - 2012-07-04	RESULT - 2012-07-04
Ubuntu 11.10 64-bit	RESULT - 2012-07-04	RESULT - 2012-07-04

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Any fix you very should have the status-firefox14 flag set to verified
If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	31	8	79%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Known Issues

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

Beta 12


Builds	FTP, HTTP
ID	20120710123126
Changeset	TBD

Checklist

[DONE] Functional automated tests, synced to Litmus
[DONE] Manual tests
[DONE] Bug fix verifications (target 50%)
[ON TRACK] Security fix verifications (target 25%)
[DONE] Update tests on releasetest
[ON TRACK] Get final sign-off from Bob Moss
[ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-07-10
Windows Vista	PASS - 2012-07-10
Windows 7 32-bit	PASS - 2012-07-10
Windows 7 64-bit	PASS - 2012-07-10
Mac OS X 10.6	PASS - 2012-07-10
Mac OS X 10.7	PASS - 2012-07-10
Ubuntu 11.10 32-bit	PASS - 2012-07-10
Ubuntu 11.10 64-bit	PASS - 2012-07-10

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows 7 32-bit	Simona	PASS - 2012-07-11

Results in Litmus

Exploratory Testing

[DONE] Exploratory testing on plugins, new features etc Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	RESULT - 2012-07-10	RESULT - 2012-07-10	RESULT - 2012-07-10
Opt-in activation for plugins	Paul Silaghi	RESULT - 2012-07-10	RESULT - 2012-07-10	RESULT - 2012-07-10
Firefox migrators - Safari Migrator	Paul Silaghi	RESULT - 2012-07-10	RESULT - 2012-07-10	N/A
Pseudo-Class Lock	Ioana Budnar	RESULT - 2012-07-10	RESULT - 2012-07-10	RESULT - 2012-07-10
Support for Mouse Capture / Mouse Lock	Virgil Dicu	RESULT - 2012-07-10	RESULT - 2012-07-10	RESULT - 2012-07-10
Mac OS 10.8 Signing	Required	n/a	PASS - 2012-07-11	n/a

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-07-11
Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-07-11
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-07-11
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-07-11
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-07-11
Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-07-11
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-07-11

Spot check build after this test - PASS - VladG - 2012-07-11

Try all the menu item functions
Gestures
Plug-in compatibility
- When I try to install missing plugins (flash) from the notification bar in youtube for example, Plugin Finder Service returns the message that "No suitable plugins were found"
Top-site compatibility
Add-on compatibility
Software update

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	PASS - 2012-07-11	PASS - 2012-07-11
Windows Vista	PASS - 2012-07-11	PASS - 2012-07-11
Windows 7 32-bit	PASS - 2012-07-11	PASS - 2012-07-11
Windows 7 64-bit	PASS - 2012-07-11	PASS - 2012-07-11
Mac OS X 10.6	PASS - 2012-07-11	PASS - 2012-07-11
Mac OS 10.7	PASS - 2012-07-11	PASS - 2012-07-11
Ubuntu 11.10 32-bit	PASS - 2012-07-11	PASS - 2012-07-11
Ubuntu 11.10 64-bit	PASS - 2012-07-11	PASS - 2012-07-11

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Any fix you very should have the status-firefox14 flag set to verified
If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	36	3	92%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Full Query

ID	Summary	Priority	Status
772786	Wrong favicons displayed in the Recently Bookmarked folder	--	RESOLVED
772842	The newly opened tab is not always focused when taking a speed test from a youtube fullscreen video	--	RESOLVED
773236	On Windows Vista Flash content is white after exiting Full Screen	--	RESOLVED

3 Total; 0 Open (0%); 3 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
675645	Firefox freezes when using Pop out while in Fullscreen mode on Youtube	P3	RESOLVED
711545	Crash reports from last evening still not found	--	VERIFIED
763237	[adbe 3221904] Firefox crash in F1398665248_____________________________	--	RESOLVED
770207	On Windows the "Feedback" button is not highlighted when hovering the mouse over it	--	RESOLVED

4 Total; 0 Open (0%); 3 Resolved (75%); 1 Verified (25%);

Firefox 14.0.1 Release Candidates


Builds	FTP, HTTP
ID	20120713134347
Changeset	e5728a4e106c

Checklist

[DONE] Functional automated tests
[DONE] Manual tests
[DONE] Bug fix verifications
[ON TRACK] Security fix verifications
[DONE] Update tests on releasetest
[ON TRACK] Get final sign-off
[ON TRACK] Update tests on release

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-07-15
Windows Vista	PASS - 2012-07-15
Windows 7 32-bit	PASS - 2012-07-15
Windows 7 64-bit	PASS - 2012-07-15
Mac OS X 10.6	PASS - 2012-07-15
Mac OS X 10.7	PASS - 2012-07-15
Ubuntu 11.10 32-bit	PASS - 2012-07-15
Ubuntu 11.10 64-bit	PASS - 2012-07-15

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows 7 32-bit	Mihaela	PASS - 2012-07-16
Mac OS X 10.7.4	Ioana	PASS - 2012-07-16
Ubuntu 12.04 32-bit	Virgil	PASS - 2012-07-16

Results in Litmus

Exploratory Testing

[DONE] Exploratory testing on plugins, new features, web compatibility etc Etherpad

Feature Testing

By now we should have all features signed off.

Mac 10.8 Signing

NOTE: Test needs to be run from a clean Mac OSX 10.8 install. Please keep a vanilla image handy so you can restore once completing this test (more info).

Download the .dmg file and drag the Firefox.app file to a location on your computer
- PASS - VladG - 2012-07-16
Open a terminal window and cd to the folder Firefox.app is located
- PASS - VladG - 2012-07-16
Run codesign -vv Firefox.app in terminal, output should contain valid on disk and satisfies its Designated Requirement
- PASS - VladG - 2012-07-16
Run xattr Firefox.app in terminal, output should contain com.apple.quarantine
- PASS - VladG - 2012-07-16
Double click the Firefox.app icon, you should see a warning about the program being provided by an unknown developer or having been downloaded from the Internet
- PASS - VladG - 2012-07-16
Click "open" and browse to any website, Firefox should start and you should be able to browse
- PASS - VladG - 2012-07-16
Quit Firefox and double click the Firefox.app icon again, you should not be reprompted for the warning in step 5, firefox should start up and browse without any issues
- PASS - VladG - 2012-07-16

Spot check build after this test.

Try all the menu item functions - PASS - VladG - 2012-07-16

Gestures
Plug-in compatibility
Top-site compatibility
Add-on compatibility
Software update

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	PASS - 2012-07-15	RESULT - 2012-07-15
Windows Vista	PASS - 2012-07-15	RESULT - 2012-07-15
Windows 7 32-bit	PASS - 2012-07-15	RESULT - 2012-07-15
Windows 7 64-bit	PASS - 2012-07-15	RESULT - 2012-07-15
Mac OS X 10.6	PASS - 2012-07-15	RESULT - 2012-07-15
Mac OS 10.7	PASS - 2012-07-15	RESULT - 2012-07-15
Ubuntu 11.10 32-bit	PASS - 2012-07-15	RESULT - 2012-07-15
Ubuntu 11.10 64-bit	PASS - 2012-07-15	RESULT - 2012-07-15

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

Any fix you very should have the status-firefox14 flag set to verified
If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	36	3	92%
QA+ SG: for the Security team to verify security fixes
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

Please look through the list of QAWANTED bugs and make sure there's nothing else we can do to advance the bug.

Issues Found

Full Query

ID	Summary	Priority	Status
774212	Improve live bookmark loading indications	P5	RESOLVED
774244	When HTML5 videos end - the suggestions are no longer displayed	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
667399	RealPlayer G2 LiveConnect plugin crash	--	RESOLVED
692446	Firefox prompts to install windows media player plugin even though it is already installed	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Releases/Firefox 14/Test Plan

Contents

Firefox 14 Test Plan

Summary

Details

Features

Nightly Merge

Endurance Automation

Functional Automation

Unconfirmed Bug Triage

Aurora Merge

Updates

Manual Testing

Smoketests

Feature Sign-offs

Issues Found

Known Issues

Pre-beta features sign offs

Issues Found

Known Issues

Beta 6

Automated Tests

Manual Tests

Smoketests

Regression Testing

Exploratory Testing

Feature Testing

Mac 10.8 Signing

Update Tests

Bug Fix Verifications

QAWANTED

Issues Found

Known Issues

Beta 6 Feedback

Beta 7

Automated Tests

Manual Tests

Smoketests

Regression Testing

Exploratory Testing

Feature Testing

Mac 10.8 Signing

Update Tests

Bug Fix Verifications

QAWANTED

Issues Found

Known Issues

Beta 7 Feedback

Beta 8

Automated Tests

Manual Tests

Smoketests

Regression Testing

Exploratory Testing

Feature Testing

Mac 10.8 Signing

Update Tests

Bug Fix Verifications

QAWANTED

Issues Found

Known Issues

Feedback

Beta 9

Automated Tests

Manual Testing

Exploratory testing etherpad: https://mozqa.etherpad.mozilla.org/14b9Exploratory

Plug-ins

Top Site Compatibility

Mac OS X 10.8 Code-signing

Update Tests

Bug Verifications

Tests

Results

Known Issues

Issues Found

Known Issues

Beta 10

Automated Tests

Manual Tests

Smoketests