QA/Firefox3/TestResults/RC1

From MozillaWiki
< QA‎ | Firefox3‎ | TestResults
Jump to: navigation, search

« QA/Firefox3/TestResults

Firefox 3 RC1 Test Results

Related Pages

Links to separate test results. Some are referenced further below.

Project Release Tracking Page

Contact Info

PM: Mike Beltzner (beltzner)
Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
5/11
Code Build
5/12
QA test signoff
ETA 5/26
  • Smoketests PASS
  • FFTs PASS
  • RC1 and past focused Tests (ongoing) PASS
  • Top Sites Testing PASS
  • Top Site Automation Pass (spidertests) 60% completed
  • Top Extensions Testing PASS
  • Top Themes Testing PASS
  • Top Plugins Testing PASS
  • Update snippets from Alpha X, Beta X -> RC1 PASS betatest, PASS releasetest, PASS beta
  • P1 L10N spotchecks PASS
  • Accessibility Testing PASS
  • All P1/P2 bug verifications (ongoing) PASS
  • Testday
    • 5/16 - PASS part 1
    • 5/23 - PASS part 2
  • Release notes, Websites, live tests PASS rc1

Release Notes

See list of release notes here

Build Location and IDs

Windows


Mac

Linux

Test Coverage

  • Areas Covered:
    • Smoketests
    • FFTs
      • Use test profiles stored here.
    • Top Sites
    • Top Plugins
    • Top Themes
    • Top Extensions
    • Focused Tests
    • Update snippets from Alpha X, Beta X -> RC1
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • Bc's topsite spidertests against jemalloc/malloc builds
    • Accessibility Tests (litmus and automation)
    • P1 L10N spotchecks
    • Bug verifications
    • RC1 Testday (2 part day)
  • Areas Not Covered:
    • localization (String changes) - Localizers to cover
    • Distribution
    • Major Updates

Test Results

Types of Tests Win XP Win Vista Mac Tiger Mac Leopard Linux
Smoketest Tomcat - PASS juanb - PASS Tracy - PASS marcia - PASS Tomcat - PASS
BFT/FFT Tomcat - PASS, FFT - PASS Tomcat BFT - PASS, FFT - PASS Tracy - BFT-PASS, Juan - FFT - PASS Marcia/juanb (Leopard test suite/PPC and Intel) BFT-PASS, (Marcia/Stephend) FFT-PASS Tracy/Abillings - BFT-PASS, FFT - PASS
Updates Testing (see below) Tchung - Betatest PASS, Releasetest PASS, Beta PASS Juanb - Betatest PASS, Releasetest PASS, Beta PASS Stephend - Betatest PASS, Releasetest PASS, Beta PASS Tomcat - Betatest PASS, Releasetest PASS, Beta PASS Abillings - Betatest PASS, Releasetest PASS, Beta PASS
Top Sites Tchung - PASS N/A Tracy - PASS N/A stephend - PASS
Top Extensions Tchung - PASS N/A Stephend - PASS N/A Abillings - PASS
Top Themes Marcia - result N/A Tracy - PASS Tomcat- in progress Abillings - PASS
Top Plugins Tchung - PASS N/A tester - result N/A Stephend - PASS
l10N Pri 1 locales Tchung - PASS N/A Tchung - PASS N/A Tracy - PASS
Accessibility MarcoZ - PASS N/A N/A N/A MarcoZ - PASS
Spidertests Bob - in progress N/A Bob - in progress N/A Bob - PASS
Bug Verifications Everyone - result Everyone - result Everyone - result Everyone - result Everyone - result
Live Checks & Download Links Tomcat - PASS rc1 Tomcat - PASS rc1 Marcia - PASS rc1 Juanb - PASS rc1 Tchung - PASS rc1

Updates Testing

We will be doing updates of all Alpha and Beta users to RC1. We will be checking partial updates, complete updates, and failed status.

Will be available on betatest, releasetest, and beta channels

  • Betatest channel - PASS
  • Releasetest channel - PASS
  • Beta channel (live) - PASS

Testcases and Results are tracked here.

Top Sites Testing

Select the top 50 sites, and run through a combination of various web applications.

Testcases

  • Visit various sites on the list, verifying web compatibility
  • Detect Layout issues, Font issues, formatting problems, broken functionality
    • Some baseline comparisons is to use Fx 2.0.0.14, Safari, and Trunk nightly builds
  • Change browser font sizes, languages and verify web compatibility
  • Operate end to end functionality of a website. Examples:
    • track an auction item on ebay
    • save a book on amazon checkout
    • register for a new email account on Yahoo Mail
    • Create a document on Google docs
    • Visit a secure site and perform online banking

Top Extensions Testing

Select a few top extensions from AMO and perform the following testcases. See Fx3 Compatibility Addons List Here.

Also, this is the top 50 most popular addons voted by mozdev.

Testcases:

  • Enabling extensions and confirming web compatibility
  • Disabling or Uninstalling extensions and confirming proper messaging and behavior
  • Load about 10+ extensions, and fire them up all at once, verifying functionality and performance
  • Updating old extension versions to the latest if applicable
  • Switch profiles with different extensions, and confirm they do not get carried over.

Extensions Test Results

Top Themes Testing

Select a few top themes from AMO and perform the following testcases. See Fx3 Compatibility Addons List Here.

Some compatible themes:

  • Classic Compact


Testcases:

  • Enabling themes and confirming web compatibility
  • Disabling or Uninstalling themes and confirming proper messaging and behavior
  • Updating old theme versions to the latest if applicable
  • Switch profiles with different themes, and confirm they do not get carried over.

Themes Test Results

Top Plugins Testing

Query the top plugins supported for the web, and verify they are compatible with Firefox 3. See AMO list for recommended plugins.

Testcases

  • On a clean environment, install plugins
  • Test updating of old plugins
  • Disable plugins in Addons Manager, verify plugins stop working
  • Uninstall plugins, verify websites request a plugin to download
  • Verify websites with popular plugins:

Additional Plugin litmus cases can be found here.

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation inspection: supported locales will be ran through Minotaur, and results will be reviewed by axel and mic.

l10 Dashboard for RC1

Testcases and Results are tracked here - PASS (Tchung, Tracy)

Accessibility Testing

  • Basic functionality
    • Page rendering to screen readers testcase 5379
      • Windows: pass
      • Linux: pass
    • Feedback in dialogs:
      • Dialogs tested: Options, Page Info, Add-Ons, Downloads
        • Windows: Pass.
        • Linux: Pass with one exception: Search fields in Add-Ons and Downloads are not spoken by Orca if no text is in them. Determined that this is an Orca bug and filed Bug 533125 against it. Fixed in Orca Trunk, will be in next release. Should be relnoted!
    • Over-all keyboard navigation: Make sure keys behave as advertised.
      • Windows: Pass.
      • Linux: Pass.
    • Menu navigation: Make sure menu bars, pull-down menus, or context menus (popups for links, for example) are spoken.
      • Windows: Pass.
      • Linux: Pass.
  • Awesome Bar (testcase):
    • Name announced when moving focus to it?
      • Windows: Pass.
      • Linux: Pass.
    • Do screen readers update when typing in addresses?
      • Windows: Pass.
      • Linux: Pass.
    • Are search results tracked when down-arrowing into them?
      • Windows: Pass.
      • Linux: Pass.
  • Larry (testcase):
    • Focusable by first going to Location bar, then shift-tabbing?
      • Windows: Pass.
      • Linux: Pass.
    • Correct text announced as name for the button?
      • Windows: Pass.
      • Linux: Pass.
    • Additional info spoken when activating button?
      • Windows: Pass: All additional info shown in the Larry popup is spoken.
      • Linux: PASS. We generate the correct events that clearly indicate the presence of this important information. However, due to Orca bug 533109. Orca is currently not picking up these alerts correctly. Should be relnoted!
  • Adding bookmarks:
    • Does screen reader speak the fact that we're in the "Page bookmarked" panel?
      • Windows: Pass. Exception is Window-Eyes who currently do not pick this one up, but they are working on a solution on their end.
      • Linux: Pass.
    • Opening folders to bookmark page in:
      • Windows: Pass.
      • Linux: Pass.
    • Tags textbox spoken?
      • Windows: Pass.
      • Linux: PASS with the known Orca bug 533125 (see also Downloads and Add-ons dialogs).
    • Checking and unchecking pre-existing tags spoken?
      • Windows: Pass.
      • Linux: FAIL! State is not changed when pressing SPACEBAR on a list item.
  • Navigation inside Library dialog:
  • Search box:
    • Confirm "Search using Google" is spoken upon first focus:
      • Windows: Pass.
      • Linux: Pass.
    • Do the different search engines get spoken upon Alt+DownArrow?
      • Windows: Pass.
      • Linux: Pass.
    • Does the name for the textbox get updated when selecting a different search engine? For example: When selecting Yahoo!, does it change to "Search using Yahoo!"?
      • Windows: Pass.
      • Linux: Pass.

Topsite Spidertests

Bob will be running topsite spidertests against nightly debug malloc and jemalloc builds. The automated coverage is to catch any crashes or assertations. Coverage will be on XP, mac, and linux.

Crash Results
Asserts Results
  • ASSERTION: Someone forgot to block scripts: 'aIsSafeToFlush == nsContentUtils::IsSafeToRunScript()', file /work/mozilla/builds/1.9.0-jemalloc/mozilla/layout/base/nsPresShell.cpp, line 4505 is fairly common. I've checked a number of these and although not all are reproducible and some at least share stacks with existing bugs, I think people would be interested in the sheer number of urls I found that fired it.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for RC. Please verify them.

P1/P2 Firefox bugs - http://tinyurl.com/3bfgoa
P1/P2 Platform bugs - http://tinyurl.com/354n48

Testday

A RC1 testday will be held and call for the community support to assist with testing. Activities include executing litmus testcases, bug verifications, and focus on the new feature areas that are being implemented. More can be found on QMO.

Testday results are posted here.

Live Spot Checks

Check live site download links on all platforms. Please cover en-US, and 2 or 3 additional Tier 1 locales as well. Pay attention to the following areas:

  • Check for signed installer and applications (windows only)
  • Firstrun page, whatsnew page, Home page, Search plugin
  • Install, uninstall
  • General spotchecks and browser functionality
  • Confirm Build IDs

OS assignments:

  • XP - tchung - PASS
  • Vista - Juanb - PASS
  • Mac Tiger - stephend - PASS
  • Mac Leopard - tomcat - PASS
  • Linux - abillings - PASS

Detailed Results

Smoketests

Tchungs Results

  • Filed bug 433440 – Setting Fx to default browser on installer does not launch browser. Looks like a dupe of bug 395891 and bug 295248
  • Renom'd bug 433298 EULA cut off checkbox/buttons on bottom - unusable on small (EeePC) screens for blocking-firefox3. Linux users in this mode have a minimal way to move forward with the installer. Duped bug 433627 in the process.

Tomcats Results

  • Smoketests RC1
    • Linux
      • Build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0
      • Status: PASS - ext/themes installation need to do later, because no compatible Extension/Themes at this time
    • Windows XP
      • Build: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0 ID:2008051206
      • Status: PASS - results in Litmus
  • BFT's
    • Windows XP
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0 ID:2008051206
      • PASS
  • BFT's
    • Windows Vista
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0
      • 'PASS - Was running into a known topcrash, but could provide the places.sqlite file to dietrich
  • FFT's
    • Windows Vista
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0
      • PASS
  • FFT
    • Windows XP
      • Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0 ID:2008051206
      • PASS
  • Bugs filed
    • bug 433796 JavaScript component does not have a method named: "observe"' when calling method: [nsIObserver::observe]
    • bug 433748 Error: this._processor is null

Tracy's Results

  • Mac (Tiger) - Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0
  • Status: PASS results in Litmus

marcia's Results

  • Smoketests RC1
    • Build ID: (Mac Leopard) - Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0
    • Status: PASS
    • Bugs Filed:
    • Issues Noted
      • I get a page alert and a dialog coming down on SSL Error exceptions. Cannot find the bug but the issue is known.
      • Favicons are not imported from Safari Bug 326701

juanb's results

  • Smoketest RC1 on Vista
    • PASS

BFTs

marcia/juanb results

    • Build ID: (Mac Leopard) - Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0 and Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0
    • Some BFT areas are being run on Intel and some on PPC and in order to get broadest spectrum of platform coverage.
    • Status: Completed
    • New Bugs Filed:
      • bug 420449 (unable to drag folders in bookmarks toolbar)
      • bug 433795 (error in the console when trying to move folder)
      • bug 421775 (moving items in the bookmarks library)
      • bug 433934 (Name field of History Entries should not look like it can be edited)
      • bug 433959 (Page info for mozilla.com shows a cnn.com cookie)
      • bug 433975 (session restores to wrong session)

Tracy's Results

  • Linux
    • Mozilla/5.0 (X11; U; Linux i686; pt-BR; rv:1.9) Gecko/2008051202 Firefox/3.0
    • Status: PASS results in Litmus
  • Mac (Tiger)
    • Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0 ID:2008051202
    • Status: PASS results in Litmus
      • bug 433942 Focus lost going back from notification bar

FFTs

marcia's results

    • Top Crashes seen during testing
    • Existing Bugs seen during FFT:
    • PPC vs. Intel
      • PPC: Migration, General
      • Intel: Installation, Menu Bar, Help, Options, Location Bar

juanb's results

  • Full Functial Tests (FFTs) on Fx3rc1 Mac 10.4.11
    • Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0
    • Status: Complete for all Mac applicable test cases (93%) as of Monday 26 @6:40pm
    • Bugs found during FFT:
      • bug 434624 - safe browsing file not updated on first launch
      • bug 434832 - remove scroll bars from import wizard
      • bug 434851 - importing data from opera fails
      • bug 434869 - data not migrated from Seamonkey
      • bug 434916 - findbar "not found" sound causes crash
      • bug 434947 - redirect urls appear in location bar autocomplete
      • bug 434951 - tags not shown in autocomplete if title...
      • bug 435287 - scroll wheel scrolling does not switch search engines
      • bug 435340 - sidebar and bookmarks library behave differently
      • bug 435470 - switch in feed prefs not reflected in feed UI page
      • bug 435485 - organize tool should be disabled when selecting left pane..
      • bug 435799 - download does not resume where it left off

Tracy's results

  • Linux
    • Build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9) Gecko/2008051202 Firefox/3.0
    • Status: PASS - results in Litmus

Hendrix Feedback

  • Top Issues Reported in Hendrix as of 5/21/08
    • Facebook chat not working
    • Hotmail not working
    • Complaints from Windows users that youtube and other video content only plays for 2-3 seconds. Complaints come from both XP and Vista users.
    • I installed Flash but it does not work (This is likely the server side bug that is open - when PFS finds the missing flash plugin it goes out and get it and seemingly installs it, but actually doesn't)
    • Ebay Refresh Auction Button missing. This causes users to have to reload the page manually. Bug 412128 will be fixed after the release.
    • Report of high CPU usage on Mac and Windows.
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/TestResults/RC1&oldid=94740"