QA/Firefox3/TestResults/Beta4
Contents
Firefox 3 Beta 4 Test Results
Related Pages
Links to separate test results. Some are referenced further below.
- Update snippets Test Results
- L10n Spotchecks Test Results
- Beta 4 Testday Results
- QA Beta 4 Relnotes Tracking
- Windows Respin Tests
Contact Info
PM: Mike Beltzner (beltzner)
Build: John O'Duinn (joduinn)
QA: Tony (tchung)
Schedule
- Code Freeze
- 2/28
- Code Build
- ETA 3/4
- QA test signoff
- ETA 3/4-3/10
- Smoketests (3/4) 100% (3 testers)
- BFTs (3/4-3/5) 100% (4 testers)
- M11 and past focused Tests (ongoing)
- Plugin Regression Tests (3/5-3/6) 100%
- Update snippets from Alpha X, Beta 1,2 -> Beta 3 (3/5-3/7) 100% betatest, 100% releasetest, 100% beta (4 testers)
- P1 L10N spotchecks (3/6) 100% (1 tester)
- B4 bug verifications (ongoing)
- Beta 4 Testday (3/7) 100% (3 testers)
- Release notes, Websites, live tests (3/10) 100% (3 testers)
Release Notes
See list of release notes here
Build Location and IDs
RC1 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b4-candidates/rc1/ RC2: http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b4-candidates/rc2/
Mac
- Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4
- Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4
Linux
- Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b4) Gecko/2008030318 Firefox/3.0b4
Windows
- Rc1: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4
- Rc2: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b4) Gecko/2008030714 Firefox/3.0b4
Strategy
- Areas Covered:
- Smoketests
- BFTs
- Beta4 and past focused Tests
- Top Plugins
- Update snippets from Alpha X, Beta X -> Beta 4
- Memory Leak Testing (handled by Tomcat on debug builds)
- P1 L10N spotchecks
- Bug verifications
- Beta 4 Testday
- Areas Not Covered:
- localization (String changes) - 3rd party will cover
- Distribution
- Major Updates
Test Results
Types of Tests | Win XP | Win Vista | Mac Tiger | Mac Leopard | Linux |
---|---|---|---|---|---|
Smoketest | Tomcat - PASS | juanb - PASS | marcia - PASS | marcia - PASS | Tomcat - PASS |
BFT | Tomcat - PASS | Tomcat (Vista test suite) - PASS | juanb - PASS | Marcia (Leopard test suite/PPC and Intel) - PASS on Intel, PASS on PPC | Tracy - PASS results in Litmus |
Top Plugins Testing (see below) | stephend - PASS - results in Litmus | N/A | stephend - PASS- results in Litmus | N/A | Abillings - PASS |
Updates Testing (see below) | Tchung - Betatest PASS, Releasetest PASS, Beta PASS | Juanb - Betatest PASS, Releasetest PASS, Beta PASS | Stephend - Betatest PASS, Releasetest PASS, Beta PASS | Marcia - Betatest PASS, Releasetest PASS, Beta PASS | Abillings - Betatest PASS, Releasetest PASS, Beta PASS |
l10N Pri 1 locales | Tchung - PASS | N/A | Tchung - PASS | N/A | Tchung - PASS |
Bug Verifications | Everyone - result | Everyone - result | Everyone - result | Everyone - result | Everyone - result |
Top Plugins Testing
Query the top plugins supported for the web, and verify they are compatible with Firefox 3. See AMO list for recommended plugins.
Testcases
- On a clean environment, install plugins
- Test updating of old plugins
- Disable plugins in Addons Manager, verify plugins stop working
- Uninstall plugins, verify websites request a plugin to download
- Verify websites with popular plugins:
- www.nintendo8.com (java)
- Linux - PASS
- www.youtube.com (flash)
- Linux - PASS
- www.shockwave.com (shockwave)
- Linux - PASS
- www.apple.com/trailers (quicktime)
- Linux - PASS
- On Linux, use vlc plugin. A test clip is available.
- Linux - PASS
- www.msnbc.com, www.cnn.com (WMV, RealVideo)
- Linux - FAIL
- RealPlayer 10 crashes under Firefox 3 - Bug 421381
- On Linux, use vlc plugin for WMV. Browser detection will often block it though.
- Linux - FAIL
- www.earthwatch.org (Adobe PDF)
- Linux - PASS
- Viewpoint Media Player plugin
- Linux - N/A (not supported)
- Mac OS X:
- Filed bug 421026 on the Viewpoint plugin for Mac, and requested blocking for Firefox 3
- Windows:
- Filed bug 421030 for an nsIView::GetView crasher when reloading a demo page using Viewpoint
- www.nintendo8.com (java)
- Others like divx, virtual tour, online ads, etc..
- PASS
- Others like divx, virtual tour, online ads, etc..
Additional Plugin litmus cases can be found here.
Updates Testing
For b4, we will be doing updates of all Alpha and Beta users to Beta 4. Tests will include different locales for beta 1, but only en-US for all past versions. We will be checking partial updates, complete updates, and failed status.
Will be available on betatest, releasetest, and beta channels
- Betatest channel - PASS
- Releasetest channel - PASS
- Beta channel (live) - PASS
Testcases and Results are tracked here.
L10N P1 Spotchecks
Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.
Automation inspection: supported locales will be ran through Minotaur, and results will be reviewed by axel and mic.
Axel's Localization Status Page
P1 Locales to be tested:
- German (de) PASS
- Japanese (ja) PASS
- Spanish (es-ES) PASS
- Netherlands (nl) PASS
- Italian (it) PASS - Opened bug 420988
- Finnish (fi) PASS
- Hindi (pa-IN) PASS
- Chinese (zh-CN) PASS - Opened bug 421213, 421219
- French (fr) PASS
Testcases and Results are tracked here - PASS.
Special Respin Tests
A build issue was found late in the game, and required a respin to all the Windows builds. This was due to a late discovery that PGO or Profile Guided Optimization, was late to be included into the build bits.
A special testplan was created to verify these bits. See Testplan here.
Bug Verifications
Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.
B4 Firefox bugs - http://tinyurl.com/3y3u4u
B4 Platform bugs - http://tinyurl.com/2mqmvx
Testday
A beta 4 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.
Testday results are posted here.
Detailed Results
Enter results from testing here.
Tomcat's Results
- Smoketests
- Windows XP
- Build ID : Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4
- Status : PASS, results in Litmus, extension and theme installation could not be tested, since no extension is ready for b4 so far
- Linux (Fedora 8)
- Build id : Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9b4) Gecko/2008030318 Firefox/3.0b4 ID:2008030318
- Status : PASS
- Windows XP
marcia's Results
- Smoketests
- Mac OS X 10.5
- Build ID : Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4
- Status : PASS, results in Litmus
- Bugs filed: bug 420917
- Crashes: http://crash-stats.mozilla.com/report/index/9674e061-ea1d-11dc-b87d-001a4bd43ef6 (Bug on file already for this crash, I expect any users that upload files regularly will hit it as it is quite easy to repro)
- Mac OS X 10.5
- Mac OS X 10.4
- Build ID : Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4
- Status : PASS, results in Litmus
- Very slow tab scrolling performance with 28 tabs open
- Mac OS X 10.4
- Mac OS X 10.5 - Intel
- Ran Leopard Test Suite, Results in Litmus, PASS
- Mac OS X 10.5 - Intel
- Mac OS X 10.5 - PPC
- Ran parts of Fun Smoketest and Form Controls Test Suite, PASS
- Mac OS X 10.5 - PPC
juanb's results
- Smoketest: PASS
- Vista
- Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4
- NOTES: Filed bug 420942, bug 420946, bug 420948 all dealing with migration from IE7 on Vista
- Vista
- BFT: PASS
- Mac Tiger 10.4.11
- Notes: Found several minor bugs dealing with places, printing. Will add the bug numbers a little later...
- bug 421139, bug 421151, bug 421208, bug 421212, bug 421246
- Mac Tiger 10.4.11
Tchung's results
- L10n testing
- Opened bug 420988 - IT locale on fx3 beta 4 fails to load default RSS feed on startup
- Opened bug 421213 - Crash [@libpthread-2.6.so@0x7290] while using baidu search plugin
- Opened bug 421219 – String.localeCompare gives strange results on OS X. zh-CN shows different order of search plugins on mac versus linux/windows.
Live Spot Checks
Live site downloads for Beta4, after bits are pushed out. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:
- Check for signed installer and applications (windows only)
- Firstrun page, whatsnew page, Home page, Search plugin
- Installer, uninstall
- General spotchecks and browser functionality
Windows XP -> abillings - PASS
Windows Vista -> stephend - PASS
Mac Tiger -> Juan - PASS
Mac Leopard -> Marcia - see results below
- he, Hebrew - fails on Mac with UI problems. Sent screenshots to tchung
- fr, Frisian - PASS
- bug 417107 has been resolved
- Identifikues montimi: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; sq; rv:1.9b4) Gecko/2008030317 Firefox/3.0b4 - PASS
Linux -> stephend - PASS
Top Issues Reported in Hendrix Against Firefox 3 Beta 4
- Enter here
Top Issues in b4 in Reporter
Hendrix feedback review [marcia]
- Gmail Contact list fails to display in FF 3 Beta 4 - bug 415252
- Yahoo Mail Beta crashes