Firefox Update Mechanism

The following is an ongoing test plan detailing QA's strategy for ensuring quality of the Firefox update experience in general.

Desktop

Compatibility

The following is a list of P1 environment configurations which need to be tested.

Locales

• Top 10 locales

Add-ons

• Compatibility Report: Fx9, Fx10, Fx11
• In-house:
  • Firebug
  • Chatzilla
  • Mozmill
  • Venkman
• First-Party:
  • Adblock Plus
  • Download Statusbar
  • DownThemAll!
  • Facemoods
  • Firebug
  • Google Toolbar
  • Greasemonkey
  • Internet Download Manager
  • Mail.Ru
  • Microsoft .NET Assistant
  • Personas Plus
  • Test Pilot
  • Video DownloadHelper
  • Yahoo! Toolbar
  • Yandex.Bar
• Third-Party:
  • Ask Toolbar
  • Avast! WebRep (binary) - bundled with Avast! Anti-virus
  • AVG Safe Search (binary) - bundled with AVG Anti-virus
  • Babylon Toolbar
  • Conduit Engine
  • DivX Plus Web Player HTML5 Video Add-on (binary) - bundled with DivX Plus Web Player
  • HP Smart Web Printing Add-on (binary) - bundled with HP Printing software
  • Java Console & Java QuickStarter (binary) - bundled with Java SE
  • Kapersky URL Advisor (binary) - bundled with Kapersky Anti-virus
  • Norton IPS & Norton Toolbar (binary) - bundled with Norton Internet Security or Norton 360
  • RealPlayer Browser Record Plugin (binary) - bundled with RealPlayer software
  • Search Helper Extension (binary) - installed via Windows Update "Search Enhancement"
  • Skype Extension & Skype Toolbar (binary) - bundled with Skype software
• Install locations:
  • Profile
  • Application Folder
  • Registry

Anti-virus Software

Here is a list of anti-virus software to test.

P1 (test these first)

• ZoneAlarm
• Microsoft Security Essentials
• Bitdefender
• ESET
• Norton
• McAfee
• AVG
• Avast!
• Kapersky

P2

• TripWire
• G Data
• Agnitum Outpost
• Panda Security
• Trend Micro
• TrustPort Security
• Webroot Security
• F-Secure

P3

• Internet security software provided by your ISP

Plug-ins

Owned by Virgil Dicu (Softvision)

Status - On track for draft tests in Litmus by 2011-11-25

• Based on SUMO article
  • Adobe Flash
  • Adobe Reader
  • Adobe Shockwave
  • Apple Quicktime
  • Gecko Media Player
  • Java
  • Microsoft Silverlight
  • RealPlayer
  • VLC Media Player
  • Windows Media Player
• Sample Test Matrix: link

Use Cases

Porting of the following usecases to testcases is being tracked in this spreadsheet

Update Paths

• partial
• partial + fallback
• complete
• complete + fallback
• pave-over to upgrade
• pave-over to downgrade
• major update unadvertised
• major update advertised
• pause, resume, hide, cancel
• timeout and notifications
• antivirus interaction (ZoneAlarm Forcefield, Symantec, McAfee, Microsoft, Kapersky, AVG, TripWire, Mac/Linux)
• forced failure at all stages of update process
• fallback from "silent" to "noisy"

Add-ons

• Third party add-ons opt-out UI
  • coverage of various install locations
• Incompatible add-ons
  • force compatible
  • outdated without an update downloaded
  • outdated with an update downloaded
  • disabled/enabled
• Compatible add-ons
  • outdated without an update downloaded
  • outdated with an update downloaded
  • updated already
  • disabled/enabled
• Personas and Themes

Session Store

• App tabs
• Panorama
• Private Browsing
• Recently closed windows
• Recently closed tabs
• Form data
  • composing an email in gmail
  • filling out user account forms
• Passwords & authenticated sessions
• Video and Audio streaming
• Shopping cart data
• Games
• User settings (e.g. Clear history)
• Upgrade right after a session restore

Places

• Bookmarks
• History

Litmus Tests

The following testcases are already in Litmus, QA's manual testing suite

Mozmill Tests

The following tests are already in Mozmill, QA's automated testing repository

• testDirectUpdate
  • test1.js: enables update logging
  • test2.js: checks for and downloads update via the About dialog
  • test3.js: checks no more updates are offered, the downloaded update was applied, and the About dialog references the correct version
• testFallbackUpdate
  • test1.js: enables update logging
  • test2.js: checks for and downloads the update via the About dialog
  • test3.js: forces a failure
    • if a complete update: checks for the error page, downloads the update again via the About dialog
    • if a partial update: uses the failure wizard to download the complete update
  • test4.js: checks no more updates are offered, the downloaded update was applied, and the About dialog references the correct version

Mobile