QA/Firefox Updates/General

Firefox Update Mechanism

The following is an ongoing test plan detailing QA's strategy for ensuring quality of the Firefox update experience in general.

Desktop

Compatibility

The following is a list of P1 environment configurations which need to be tested.

Locales

• Top 10 locales

Add-ons

• Compatibility Report: Fx9, Fx10, Fx11
• In-house:
  • Firebug
  • Chatzilla
  • Mozmill
  • Venkman
• First-Party:
  • Adblock Plus
  • Download Statusbar
  • DownThemAll!
  • Facemoods
  • Firebug
  • Google Toolbar
  • Greasemonkey
  • Internet Download Manager
  • Mail.Ru
  • Microsoft .NET Assistant
  • Personas Plus
  • Test Pilot
  • Video DownloadHelper
  • Yahoo! Toolbar
  • Yandex.Bar
• Third-Party:
  • Ask Toolbar
  • Avast! WebRep (binary) - bundled with Avast! Anti-virus
  • AVG Safe Search (binary) - bundled with AVG Anti-virus
  • Babylon Toolbar
  • Conduit Engine
  • DivX Plus Web Player HTML5 Video Add-on (binary) - bundled with DivX Plus Web Player
  • HP Smart Web Printing Add-on (binary) - bundled with HP Printing software
  • Java Console & Java QuickStarter (binary) - bundled with Java SE
  • Kapersky URL Advisor (binary) - bundled with Kapersky Anti-virus
  • Norton IPS & Norton Toolbar (binary) - bundled with Norton Internet Security or Norton 360
  • RealPlayer Browser Record Plugin (binary) - bundled with RealPlayer software
  • Search Helper Extension (binary) - installed via Windows Update "Search Enhancement"
  • Skype Extension & Skype Toolbar (binary) - bundled with Skype software
• Install locations:
  • Profile
  • Application Folder
  • Registry

Anti-virus Software

Here is a list of anti-virus software to test.

P1 (test these first)

• ZoneAlarm
• Microsoft Security Essentials
• Bitdefender
• ESET
• Norton
• McAfee
• AVG
• Avast!
• Kapersky

P2

• TripWire
• G Data
• Agnitum Outpost
• Panda Security
• Trend Micro
• TrustPort Security
• Webroot Security
• F-Secure

P3

• Internet security software provided by your ISP

Plug-ins

Owned by Virgil Dicu (Softvision)

Status - On track for draft tests in Litmus by 2011-11-25

• Based on SUMO article
  • Adobe Flash
  • Adobe Reader
  • Adobe Shockwave
  • Apple Quicktime
  • Gecko Media Player
  • Java
  • Microsoft Silverlight
  • RealPlayer
  • VLC Media Player
  • Windows Media Player
• Sample Test Matrix: link

Use Cases

Porting of the following usecases to testcases is being tracked in this spreadsheet

Update Paths

• partial
• partial + fallback
• complete
• complete + fallback
• pave-over to upgrade
• pave-over to downgrade
• major update unadvertised
• major update advertised
• pause, resume, hide, cancel
• timeout and notifications
• antivirus interaction (ZoneAlarm Forcefield, Symantec, McAfee, Microsoft, Kapersky, AVG, TripWire, Mac/Linux)
• forced failure at all stages of update process
• fallback from "silent" to "noisy"

Add-ons

• Third party add-ons opt-out UI
  • coverage of various install locations
• Incompatible add-ons
  • force compatible
  • outdated without an update downloaded
  • outdated with an update downloaded
  • disabled/enabled
• Compatible add-ons
  • outdated without an update downloaded
  • outdated with an update downloaded
  • updated already
  • disabled/enabled
• Personas and Themes

Session Store

• App tabs
• Panorama
• Private Browsing
• Recently closed windows
• Recently closed tabs
• Form data
  • composing an email in gmail
  • filling out user account forms
• Passwords & authenticated sessions
• Video and Audio streaming
• Shopping cart data
• Games
• User settings (e.g. Clear history)
• Upgrade right after a session restore

Places

• Bookmarks
• History

Litmus Tests

The following testcases are already in Litmus, QA's manual testing suite

Test ID	Summary
3.6 -> 8.0 Major Update
31850	Check for updates and verify a major update is offered
31854	Selecting "Details" in "Downloading Update" opens new window on update details page
31858	Selecting "Get the new version" in Software Update dialog takes user to "Downloading Update"
31859	Selecting "Never" in Major Update details dialog dismisses it forever
31857	Selecting "Later" in Major Update details dialog dismisses it
31851	Major Update details are in locale language
31861	Selecting "Restart" in "Downloading Update" after download is complete applies update
32145	Verify navigation buttons appear correctly
33512	Upgrade Firefox while having add-ons pending for update installation
31860	Selecting "Pause" in "Downloading Update" pauses update downloading
31848	After update, compatible extensions still work
31849	After update, compatible themes still work
31852	On first run after install, users are warned about incompatible extensions and can update them
31853	On first run after install, users are warned about incompatible themes and can update them
31862	Selecting "Resume" in "Downloading Update" of paused update resumes download
31863	Users are warned about incompatible extensions
31864	Users are warned about incompatible extensions and can cancel updating them
31865	Users are warned about incompatible extensions and ignore the warning
31866	Users are warned about incompatible themes
31867	Users are warned about incompatible themes and can cancel updating them
31868	Users are warned about incompatible themes and ignore the warning
31855	Selecting "Hide" in "Downloading Update" hides dialog
31856	Selecting "Hide" in "Downloading Update" with paused download prompts user
32505	Users are not warned about disabled incompatible extensions
32853	Verify extensions present on downgrade to previous Firefox version
33512	Upgrade Firefox while having add-ons pending for update installation
Software Update
31792	Apply software update should display Restart Confirmation
31803	Software update dialog wait for idle before prompting
31287	No Updates Found
31492	Make sure we don't offer the same update again
31648	Make sure users that are not Member of the Administrator group don`t receive Update Notifications
32856	Make sure that auto-detect proxy settings do not interfere with software updates

Mozmill Tests

The following tests are already in Mozmill, QA's automated testing repository

• testDirectUpdate
  • test1.js: enables update logging
  • test2.js: checks for and downloads update via the About dialog
  • test3.js: checks no more updates are offered, the downloaded update was applied, and the About dialog references the correct version
• testFallbackUpdate
  • test1.js: enables update logging
  • test2.js: checks for and downloads the update via the About dialog
  • test3.js: forces a failure
    • if a complete update: checks for the error page, downloads the update again via the About dialog
    • if a partial update: uses the failure wizard to download the complete update
  • test4.js: checks no more updates are offered, the downloaded update was applied, and the About dialog references the correct version