The following documents testing of graphics in Firefox 38 across a wide range of test environments. For more information on the methodology please consult our wiki.

Suggested Systems

Run tests once with the latest GPU driver available and a second time with whichever older driver version you have access.

What to look for

Videos:

• significant buffering pauses
• black video
• spikes in CPU and memory usage
• crashes and/or hangs
• tab rendering issues

WebGL/Canvas2d:

• crashes
• hangs
• visible artifacts
• black screen
• complete inability to load the demo(s)
• high cpu/memory usage

Instructions

Setting up the test environment

1. Install Firefox

• Download https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/firefox-38.0a2.en-US.win32.installer.exe
• Run installer and select customize
• Put the install in a known directory (C:\Aurora for purposes of this example)
• Do NOT allow it to launch Firefox at the end of the install

2. Disable the automatic updater

• Open C:\Aurora\defaults\pref\channel-prefs.js in a text editor
• Change the line reading pref("app.update.channel", "release"); to pref("app.update.channel", "foobar");
• Save and Close the text file

3. Launch Firefox with a clean profile

• Run cmd.exe
• Change directory to the folder where you installed Firefox: cd C:\Aurora
• Run firefox.exe: firefox.exe -P -no-remote - This will bring up our profile selector window.
• Use the "Create Profile" button to create a profile new profile
• Once the profile is created, select it, and then click "Start Firefox"

4. Load some videos in the background [Windows Vista/7/8 only]

• Open each of the following videos in new tabs
  • https://www.youtube.com/watch?v=Jp59FElhOTw
  • https://www.youtube.com/watch?v=79ImZE0K7xc
  • https://www.youtube.com/watch?v=7SRTEXSpcyI
• Set the quality of these videos to the highest (1080p60 or 4K)
• Make sure these videos run in the background for the duration of the testrun

Reporting a crash

If you encounter a crash you will see a crash reporter dialog. Use this to report the crash to Mozilla.

• Enter a comment to describe the circumstances of the crash
• Click submit on the crash reporter dialog
• Restart Firefox
• Load the about:support page and copy all of the information on that page to a text file, upload it to the Google drive location
• Load the about:crashes page and copy the link of the crash report you submitted (it will be the one at the top of the list)
• Report the link to the about:support page in Google drive as well as the link to your crash report

Tests

Note: Systems running Windows XP do NOT run test 1-4, start on test 5

1. Crash test [Windows Vista/7/8 only]

• Load each of these URLs in new tabs:
  • http://www.detik.com/
  • https://www.irctc.co.in/eticketing/home
  • http://www.prothom-alo.com/
• Resize the window smaller and larger to random sizes
• Click a link on the page and resize the window some more
• Switch into responsive design view (CTRL + Shift + M), change resolution, and refresh the page
• If you encounter a crash then proceed with reporting a crash instructions
• If you do NOT encounter a crash then this test is a PASS, move on to the next test

2. Basic test [Windows Vista/7/8 only]

• Load the about:config and click I'll be careful.
• Search for dxva and set media.windows-media-foundation.use-dxva to true then restart Firefox
• Load https://www.youtube.com/watch?v=hicBgE6XndM
• Use gear icon to set to highest quality (1080HD)
• Open full screen and play video
• Use the bottom bar of the video to reset different quality levels (using gear icon)
• Use the arrow keys (or the slider at the bottom of the page) to seek forward and back through the video
• Make at least three changes to quality level and seeking then stop the video
• Close both tabs

3. Stress test [Windows Vista/7/8 only]

• Load the following URLs each in their own tab:
  • https://www.youtube.com/watch?v=hicBgE6XndM
  • https://www.youtube.com/watch?v=mGRpobfnQ8k
  • https://www.youtube.com/watch?v=mpi0qsp3v_w
  • https://www.youtube.com/watch?v=xdWCoEl3VRk
  • https://www.youtube.com/watch?v=sC4n9BcwkGQ
  • https://www.youtube.com/watch?v=s2REEePe3Mk
  • https://www.youtube.com/watch?v=qqPX1t-98rU&hd=1
  • https://www.youtube.com/watch?v=e-ORhEE9VVg
• Cycle through the tabs, setting all videos to theater mode
• Cycle through the tabs, setting all videos to highest bitrate playback mode (using the gear icon)
• Cycle through the tabs, seeking all videos backward
• Cycle through the tabs, setting all videos to a 480p video mode and seek them forward
• Cycle back through the tabs, set to highest bitrate and seek backward far enough that they have at least 30s remaining on all videos
• Let everything play at once for about 30s

4. Test with system sleep [Windows Vista/7/8 only]

• Load https://www.youtube.com/watch?v=hicBgE6XndM
• Use gear icon to set to highest quality (1080HD)
• Let the video play to about halfway
• Put your system into sleep mode
• Restore your system and resume watching the video

5. Canvas2d demo [All systems]

• Load http://ie.microsoft.com/testdrive/HTML5/Blizzard/Default.html
• Check to see that snowflakes animate on the screen
• NOTE: Slow performance is acceptable - we're just concerned if the flakes show or not

6. Once you've completed the test run... [All systems]

• Save a copy of about:memory:
  • Open a tab to about:memory
  • Click measure and save, use the naming convention fx38_<machinename>_<driverlevel>_memory.json
  • Upload this file to the Graphics Test Reporting/Firefox 38/Memory Reports/ folder on Google drive
  • Click both the GC and CC buttons to free up memory
• Report any bugs you encountered
• Fill out this form to report your results

Results

Notes

• bugs
• test plan was well described
• managed to get through the testing faster than planned
• should we ask for regression range using mozregression out of the gate?
  • betabreakers could use better documentation on how to find a range with mozregression and finding the build date
• finding median driver versions is a challenge depending on how discoverable they are on the web
• remote access for system troubleshooting may be possible
• similar failures should be filed in a single bug and then split out of they're found to be different issues through investigation
• what is the expectation for engaging Betabreakers for follow-ups?
  • we should seek to get as much information upfront to minimize the follow-up cycle
  • follow-up is possible but may be delayed if the tester has moved on to another project
  • we should account for a percentage of follow-up time in future contracts
• include systems with blacklisted drivers in future testing
• may need a standard list of hardware that we always use
• Is betabreakers set up with Windows 10?
  • they do but not in the general tester population
  • look into doing a testrun later in the summer
• how representative of our Beta audience is Betabreakers?

Actions

• Anthony to provide better documentation for mozregression and getting a build ID
• Anthony to provide documentation on attaching about:support, fhr, and telemetry to bug reports
• Jeremy will follow up with IT to figure out future Win10 testing capability

Correlations

Operating Systems:

• 5 bugs against Windows Vista
• 3 bugs against Windows 8.1
• 2 bugs against Windows 7
• 1 bug against Windows XP

GPUs:

• 7 bugs against NVidia
• 4 bugs against AMD
• 0 bugs against Intel

Architectures:

• 7 bugs against 32-bit
• 4 bugs against 64-bit

Processors:

• 8 bugs against Intel
• 3 bugs against AMD