MozillaQualityAssurance:Environments
This page explores some of the topics around creating Firefox test environments that mimic "real world" and "typical" end user configurations.
Contents
- 1 Overall Goal
- 2 Requirements
- 3 Recommendation
- 4 Notes from Profile and Configuration Meeting Held on 6-13-07
- 5 Win XP Profile Data
- 6 Location of Information
- 7 Notes from Profile and Configuration Meeting Held on 6-21-07
- 8 Next Steps/Issues
- 9 Target Completion Date for Phase 1 Implementation
- 10 Profiles created in December, 2007 by Platform
- 11 Most used Add-on 's
Overall Goal
Configure the Mozilla testing systems as close as possible to a "typical" Firefox user in order to discover problems early and provide better configuration coverage.
Every user does not install Firefox into a fresh copy of Windows XP SP2 and people do not always upgrade to the latest and great MSFT patches. Their profiles may have been used for years and they have their web cache full of data or run a bunch of extensions. The goal of this exercise is to come up with a reasonable set of configs that best represent some typical configurations of our users.
Our current testing methodology uses a (random? chaotic?) combination of configuration environments that are not documented, controlled or reproducible.
Requirements
- Scope out several "typical end user configurations" for Firefox.
- Offer an easy way to do testing and throw away the environment
- Keep the base environments up to date
- Easily "apply a specific Firefox config" against the base environment
- Some of the parameters to consider include:
- Operating System (clean install vs. existing installation), version number, patches, etc... - Windows XP, Vista, Mac, Linux (which one?)
- OS Languages (e.g. are they running US Windows XP or Japanese XP)
- Firefox (clean new install), no profile versus Existing Profile
- Profile character:
- Clean web cache vs. 125 MB worth of stuff
- 2 vs. 15 addons (which ones to select?)
- Selection of saved passwords?
- Other profile aspects?
- One versus multiple OS users? (not sure)
- One versus multiple Firefox profiles? (not sure)
Recommendation
- Develop a "minimum set" (3-4 configs) and an "expanded set" (up to 10?)
- Use our Vmware server and "snapshot capabilities" to define the following sets of VM's
- Typical Profiles
- No Profile (ie, new user to Firefox)
- Profile A: 120 MB worth of cached web pages, 3 addons [typical]
- Profile B: 200 MB worth of cached web pages, non-default home page, 10 addons [stress test]
- Typical OS Setups
- OS Setup 1: Windows XP SP2 + all high priority WindowsUpdate patches
- OS Setup 2: Windows XP SP2 + no additional patches
- OS Setup 3: Mac OS X 10.4.9 + all available SoftwareUpdate patches
- Minimum Set:
- Config 1: OS Setup 1 + no Firefox Profile
- Config 2: OS Setup 1 + Firefox 2.0.0.x + Profile A
- Config 3: OS Setup 1 + Firefox 2.0.0.x + Profile B
- Config 4: OS Setup 3 + Firefox 2.0.0.x + Profile A
- Expanded Set:
- Config 5: OS Setup 3 + Firefox 2.0.0.x + Profile B
- Config 6: OS Setup 3 + no Firefox Profile
- Config 7: OS Setup 2 + no Firefox Profile
- Create VM instances for OS Setup 1 sn
- Use the VM instances in "read only mode" to test
- VM Instance maintenance: Periodically, update the base config with the latest Microsoft Windows patches
- Our Vmware Server allows us to store multiple copies of vm instances. So, perhaps we create several base instance (e.g. Windows XPSP2, Windows XP SP2+all current patches) and then script the creation of getting these base instances to the configs that we want (e.g. loading them with profile data, etc...).
- For Mac based testing of Window Firefox, you can do the same thing but using Parallels. Create a clean Windows XPSP2 instance, save the config and copy it to a file server location. Using a copy of that instance, create Windows XPSP2+patches, save that instance. When testing Firefox, use a copy from the fileserver and then throw away the instance.
- Keep the base instances updated with the latest patches from Microsoft periodically.
- For Mac testing, not certain how we do that.
Notes from Profile and Configuration Meeting Held on 6-13-07
- Attendees: timr, juanb, tchung, marcia, tomcat, galen
Typical OS Setup:
Tier 1:
- Windows XP with latest updates
- Windows Vista with latest updates
- Windows 2K with latest updates
- Mac OSX 10.4.x with latest updates (PPC and Intel)
- Linux - Ubuntu and Red Hat
- First implementation will focus on Tier 1 only
- First implementation will focus on en-US only
Tier 2:
- XP SP2 - No patches
Profile A | Profile B | Profile C | Profile D | |
---|---|---|---|---|
1. Memory Cache | 7500 KiB | 25000 KiB | 50000 KiB (maximum) | |
1. History | bc data | bc data | ||
1. Cookies | bc data | bc data | ||
1. Bookmarks | 20 | 100 | 1000 | |
1. Downloads | 10 | 100 | 1000 | |
1. Form Data | bc data | bc data | ||
1. Passwords | 5-10 | 20 | 100 | |
1. Addons | 5 | 25 | 50 | |
1. Plugins (Java, Flash, etc) | Flash, Java | Flash, Java, Acrobat, Platform dependent media player | All above plus Viewpoint | |
2. Tabs | 2 | 1 | 5-10 | 25 |
2. Non Default Homepage | ||||
2. Installation History | ||||
2. Multi Users | ||||
3. Search Plugins | 0 | 5 | 10 | |
3. Preferences | Does not change the prefs |
Win XP Profile Data
Win XP Profile A - Branch (created with 2.0.0.6) | Win XP Profile B - Branch (created with 2.0.0.6) | Win XP Profile C - Branch (created with 2.0.0.6) | Win XP Profile D - Branch (created with 2.0.0.6) | |
---|---|---|---|---|
1. Memory Cache | 4217 KiB (from about:cache) | |||
1. History | 117KB | |||
1. Cookies | 25 KB | |||
1. Bookmarks | 20, saved in Folder in Bookmarks directory. 1 RSS feed saved in Toolbar. File size=15KB | |||
1. Downloads | downloads.rdf=10 KB | |||
1. Form Data | Accumulated from password saves | |||
1. Passwords | 5 passwords saved | |||
1. Addons | 6 Total: Adblock Plus, Forecast Fox Enhanced, Flashgot, Download Statusbar, IE Tab, Tab Mix Plus | |||
1. Plugins (Java, Flash, etc) | Flash verion 9.0 r47, Java version 1.6.0._02 | |||
2. Tabs | Default tabs | |||
2. Non Default Homepage | Default Homepage | |||
2. Installation History | Installed to default directory | |||
2. Multi Users | None | |||
3. Search Plugins | Default | |||
3. Preferences | Does not change the prefs |
Priority ranking of items
1Cache
1History
1Bookmarks
1Downloads
1Form Data
1Passwords
1Extensions and Themes
2Tabs
2Non Default Homepage
1Plugins (Java, Flash)
3Search Plugins
2Installation History - Clean Install/Paveover Install/Upgrade
2Multi-Users User 2 users Admin/Non Admin 2+Users
3Preferences
Location of Information
- Disk Cache
- (Example) /Users/marcia/Library/Application Support/Firefox/Profiles/5aym641t.2004/Cache
- Type about:cache in the browser to see your disk and memory cache
- History
- 1.8 branch=history.dat
- 1.9 branch=places.sqlite
- Form History
- 1.8 branch=formhistory.dat
- 1.9 branch=formhistory.sqlite
- Search
- 1.8 branch=search.sqlite
- 1.9 branch=search.sqlite
- Bookmarks
- 1.8 branch=bookmarks.html
- 1.9 branch=places.sqlite
- Downloads
- 1.8 branch=downloads.rdf
- 1.9 branch=downloads.sqlite
- Cookies
- 1.8 branch=cookies.txt
- 1.9 branch=cookies.sqlite
Profiles
- A No Profile
- B Typical Profile
- C Large Profile
- D Stress Profile
Configurations
- 1 OS1 - FF 2.0.0.x - Profile A (Smoke)
- 2. OS1 - FF 2.0.0.x - Profile B (BFT)
- 3. OS1 - FF 2.0.0.x - Profile C (Smoke)
- 4. OS2 - FF 2.0.0.x - Profile B (Smoke)
- 5. OS3 - FF 2.0.0.x - Profile B
- 6. OS4 - FF 2.0.0.x - Profile B
Other Issues
- 110nLocales
- Admin versus Non Admin
- Priorities: Magnitude of problem (data loss)
- Common Compatibility (Problems with Java)
Notes from Profile and Configuration Meeting Held on 6-21-07
attendees: marcia, tomcat and bc
Action Items from meeting
- Tomcat: Talk to Shaver about Popular extensions on AMO
- Review Asa blog post - Extensions you use-may be some data points there
- Follow up with Jay poll survey capability on QMO->marcia
- We may need a separate one for "How many extensions" and then 'What extension?"
- Determine where relevant data is stored in FF 2 and FF3->marcia
- Profile D->Data from 15K page run->bc
Discussion Points/Recommendations
- Addon Selections
- NoScript
- Adblock
- Flashblock
- Number of Addons installed
- A No Profile
- B Typical Profile -5-10
- C Large Profile - 25
- D Stress Profile -50
- Page cache->50 is the
- Downloads->
- Form data->formhistory.dat
- Sessionstore.js
- Sessonstore1.js
- Sessionstore.bak
- Missing from the Chart
- Search History
- Certificates
- RSS feeds and Microsummaries
- Tabs
- 2, 1, 5-10, 25
- Passwords
- 5-10 Typical, 20, 100
- Bookmarks
- 20, 100, 1000
- Downloads
- 10, 100, 1000
- Search Plugin
- 0, 5, 10
- Plugins
- Typical->Flash and Java
- C->Flash, Java, Acrobat, Media Player depending on platform, Java
- D->Every one including Viewpoint.
Issues
- The stress test profiles generated by bc don't seem to launch on Win XP.
- Need to ensure our Machine installation for Java-match what we end up in our installation
- Determine Platforms differences that will affect profiles
- Mac Plugins used can be different
- Mac Add ons may not work with other profiles
- What goes into the Installation Directory for Plugins
- On Linux, they are stored in a special place on their profile
- Need to realize that we need a set of profiles specific to FF 2 and FF3, and that data stored between the two versions will be stored in different places.
Next Steps/Issues
- Solicit feedback from QA team and others as to what data makes sense. We need help determining how to extract the relevant data from the browser.
- Recommendation made to put a survey up on QMO to gather data
- Investigate an easy way to get addons popularity data. We have a good sense of what are the popular extensions, but conflicting data from sites such as extensionhunter.com make it difficult to assess-perhaps morgamic can help here.
- Need some help building out the complex profiles (simulating things like 10K downloads). Who from QA/dev can assist here?
VM Issues/Followup
- Follow up needed to see where Justin is with VMware investigation
- Follow up needed for Mac VM Investigation->Someone should try out PearPC (http://pearpc.sourceforge.net/) and see how well it works. It lets you run Mac OS X on a PC
- Discussion with josh, I think the feeling overall is that we are not going to pursue PearPC. We need to make sure we have balance of PPC/Intel machines to run.
Target Completion Date for Phase 1 Implementation
- Would like to see a set of 4 profiles created in time for testing of the Alpha6 release
Profiles created in December, 2007 by Platform
Profile B
PPC Mac 10.4 Profile | Intel Mac 10.5 Profile | Win Vista Profile | Win XP Profile | Linux Ubuntu Profile | ||
---|---|---|---|---|---|---|
1. Memory Cache | 9142 KiB | 31950 KiB | 13986 KiB | 99 KiB | ||
1. History | ||||||
1. Cookies | ||||||
1. Bookmarks | 20 | 20 | 20 | 20 | ||
1. Downloads | 10 | 10 | 10 | 10 | ||
1. Form Data | ||||||
1. Passwords | 5 | 5 | 5 | 5 | ||
1. Addons | 5 Extensions and 2 Themes | 5 Extensions and 2 Themes | 5 Extensions and 2 Themes | 5 Extensions and 2 Themes | ||
1. Plugins (Java, Flash, etc) | Flash r115, JEP 0.9.6.3 | Flash r115, Java Java Plug-in 1.6.0_03 | Flash r115, Java Java Plug-in 1.6.0_03 | Flash r115, Java install pending | ||
2. Tabs | ||||||
2. Non Default Homepage | ||||||
2. Installation History | ||||||
2. Multi Users | ||||||
3. Search Plugins | ||||||
3. Preferences |
Machine Specs that Profiles were created on:
- Win Vista Home Premium
- Intel Duel CPU 2.93 Ghz with 3326 RAM
- Win XP VM Image
- Created with the VM in the QA virtualization directory
- Intel Mac 10.5
- 2.2 Ghz Intel Core Duo with 3 GB RAM
- Ubuntu
- Created with the VM in the QA virtualization directory
Most used Add-on 's
- Flashgot
- No Script
- Noia 2.0 extreme Theme
- Video Downloader
- Adblock +
- Forcastfox
- Fasterfox
- Adblock
- IE Tab
- Download Statusbar
- Downloadthem all
- US-dict
- StumbleUpon
- Blue Ice Theme
- Customize Google
- PDF Download
- Foxytunes
- Dictionnaire MySpell en Français (réforme 1990)
- Tabbrowser Prefs
- Tabmix Plus
- Webdeveloper
- Fireftp
- Silver Skin Theme
- Firefox Vista Theme
- German Dict
- RedShift V2 RC1 Theme
- Greasemonkey
- Spanish Dict
- Metal Lion Vista Theme
- Firebug
- Foxmarks
- Firebug
- Adblock Filterset
- my Firefox Theme
- Blueshift Theme
- en-GB Dict
- Chatzilla
- Qute Theme
- Video Download Helper
- Imagezoom
- Cooliris Previews
- Mouse Gestures
- i Fox Theme
- Vista Aero