QA/Firefox3/TestResults/M9

From MozillaWiki
Jump to: navigation, search

« QA/Firefox3/TestResults

Firefox 3 M9 Test Results

Related Pages

Most of these pages are linked below in this document.

Contact Info

Build: John O'Duinn (joduinn)
QA: Tony (tchung)

Schedule

Code Freeze
11/2
Code Build
11/7
QA test signoff
ETA 11/13

Release Notes

Build Location and IDs

RC1 builds

  • Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1
  • Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
  • XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
  • Linux Build ID: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110618 Firefox/3.0b1

RC3 builds

  • Mac Build ID: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
  • Vista Build ID: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1
  • Linux Build Id: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9b1) Gecko/2007110903 Firefox/3.0b1
  • XP Build ID: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1

Features landed

Firefox

  • List coming soon

Gecko & Platform

  • List coming soon

Strategy

  • Areas Covered:
    • Smoketests
    • FFTs
    • M9 and past focused Tests
    • Top Extensions
    • Top Plugins
    • Top Sites (manual by QAE, automation by bc)
    • Memory Leak Testing (handled by Tomcat on debug builds)
    • Lite L10N spotchecks only. (need list of supported locales)
    • M9 bug verifications
  • Areas Not Covered:
    • L10N
    • Distribution
    • Updates

Tests are ran against en-US locale only.

Test Results

Types of Tests Win XP Win Vista (include Vista Litmus Test Suite) Mac Tiger Mac Leopard Linux
Smoketest Tomcat - PASS Tomcat - PASS Tracy - PASS N/A Stephend - PASS
FFT Tomcat - PASS Tomcat (partial) - PASS Tchung - PASS Marcia - PASS Tomcat - PASS
Top Sites Testing (see below) Juanb - PASS
bc (automation) - results 		N/A Juanb - PASS N/A bc (automation) - results
Top Extensions Testing (see below) Abillings - PASS N/A Abillings - PASS N/A Tracy - PASS (only Adblock Plus and Gmail nofication compatible)
Top Plugins Testing (see below) Stephend - PASS N/A Stephend (Marcia, really, in her FFT testing; thanks Marcia!) - PASS N/A Tracy - PASS (only able to install Flash and PDF)
l10N Pri 1 locales Tracy (de, en-GB, pl, fr) - PASS, en-GB is n/a - Tomcat (DE) PASS (one regression found, see results) N/A Tchung (ja, es-ES) - PASS N/A N/A
Bug Verifications Everyone - PASS Everyone - PASS Everyone - PASS Everyone - PASS Everyone - PASS
  • Windows XP Smoketest, en-US, - Tomcat
    • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110703 Firefox/3.0b1
    • PASS - addons testing currently not possible, because of no addons ready for maxversion b1

Top Sites Testing

Select the top 50 sites, and run through a combination of various web applications.

Testcases

  • Visit various sites on the list, verifying web compatibility
  • Detect Layout issues, Font issues, formatting problems, broken functionality
    • Some baseline comparisons is to use Fx 2.0.0.9, Safari, and Trunk nightly builds
  • Change browser font sizes, languages and verify web compatibility
  • Operate end to end functionality of a website. Examples:
    • track an auction item on ebay
    • save a book on amazon checkout
    • register for a new email account on Yahoo Mail
    • Create a document on Google docs
    • Visit a secure site and perform online banking

Top Extensions Testing

Select a few top extensions from this list and run tests to install/update/uninstall/disable extensions. Available for download at AMO.

Or, refer to this list: Most-Used Addons

Testcases:

  • Enabling extensions and confirming web compatibility
  • Disabling or Uninstalling extensions and confirming proper messaging and behavior
  • Load about 10+ extensions, and fire them up all at once, verifying functionality and performance
  • Updating old extension versions to the latest if applicable
  • Switch profiles with different extensions, and confirm they do not get carried over.

Extensions Test Results

Top Plugins Testing

Select a few top plugins on the web. Use various websites supporting these plugins. Available for download at AMO.

Consider:

  • Adobe PDF
  • Flash
  • Java
  • Shockwave
  • Embedded Win Player
  • Embedded Divx player
  • Quicktime player
  • Viewpoint on both Vista and XP
  • Ipix (360 degree view)

Testcases:

  • Enabling Plugins and confirming web compatibility
  • Disabling Plugins and confirming proper messaging and behavior
  • Check about:plugins and Addon Managers to verify plugins still exist
  • Updating old plugin versions to the latest

Issue found:

Pre-existing bug 328675

L10N P1 Spotchecks

Visual inspection: Go through the P1 list of locales, and run a spotcheck to see if everything visually looks fine. Refer to L10N Requirements for more specifics.

Automation: We'll use 2.0.0.9 to create the Minotaur reference files.

P1 Locales to be tested:

  • English (en-GB) not available, localizer not ready
  • French (fr) PASS
  • German (de) PASS
  • Japanese (ja) PASS
  • Polish (pl) PASS
  • Spanish (es-ES) PASS

I put all the beta1 available P1's through a Minotaur check against 2.0.0.9. That looked good; nothing glaringly bad jumped out at me. However, I need to get those files loaded up somewhere so Axel and Mic can have a look.

Testcases and Results are tracked QA/Firefox3/TestResults/M9/L10N.

Bug Verifications

Bugs that are marked RESOLVED FIXED that are checked in for beta. Please verify them.

Detailed Results

Tchung Results
marcia's leopard bugs and problem areas
Tomcat Results
  • FFT`s PASS
    • Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1
    • Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b1) Gecko/2007110904 Firefox/3.0b1

Focus testing Windows / Windows Vista : PASS

    • Bugs filed:
      • bug 403497 Places Toolbar Folder not renamed when language version is changed
      • bug 403401 Wrong Link to Store "Firefox on CD" on www.mozilla.org
      • bug 403257 Localized Extension description not maintained when the Add-on is disabled
      • bug 403246 Last line of Anti Phishing Warning Page contains de and en-US text
      • bug 403185 "Open All in Tabs" appends, rather than replaces, current tabset
      • bug 403628 Failed to load module "gnomebreakpad - Error output on Trunk Start with Fedora 8 -
      • bug 392919Breakpad System itself is broken on Fedora 8 - this should block a beta 2 i guess
      • bug 403842 Default RSS Feed (Aktuelle Nachrichten) fails to load on DE Firefox 3 Beta 1 RC - Dupe of Bug 381357
      • bug 404081 "Warn me when i close multiple tabs" Pref can be overridden
      • bug 404074 Clear Private Data Menu is cut off in french firefox 3 beta 1
      • bug 403937 Missing title on Mac for "about Mozilla Firefox" Window


  • DE Locale Spot Check - PASS was running into bug 403842, see above


Stephend Results
  • Prexisting bug 328675 - I nominated it for "blocking-1.9?", but jst claims plugins aren't really being actively worked on :-(
Juanb results
(topsite)
11/9/07 Testday Results

Full Summary here. Highlights:

  • 1300 litmus tests by Community
  • 500 litmus tests by l10n Community
  • 56 failed testcases
  • 60 unclear/broken testcases
  • 50 new Community Members total, and 35 using Litmus

Live Spot Checks for M9

When we go live at 3 PM it would be great team members could download an M9 build quickly and make sure everything looks ok.

  • Windows XP -> abillings - PASS
  • Windows Vista -> stephend/abillings (en-US, de) - PASS
  • Mac PPC/Intel-> tchung - PASS
  • Mac Leopard -> abillings - PASS
  • Linux -> stephend - (en-US/nl) PASS

DevTracking Bugs for M9

Refer to dev-tracking bugs at: [M9 firefox]|[M9 platform]

Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/TestResults/M9&oldid=75339"