Electrolysis/Release Criteria

From MozillaWiki
Jump to: navigation, search

Placeholder page to collect and track Electrolysis release criteria. cpeterson owns this page.

Rollout Criteria

We plan to ship e10s in Firefox 48 using a staged rollout. See elan's e10s release plan.

Release Criteria

Stability

Responsible bsmedberg
Accountable bsmedberg
Supporting Kairo, rvitillo
Consulted Barbara, release management
Informed cpeterson, elan
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
chrome + content crash rate (chrome + (content - content shutdown termination crash reports) ) / 1000 use hours "M+(c-s)" on 48 Beta jimm 5.34 crashes/khour 5.20 crashes/khour YES Barbara (2016-06-29), bsmedberg (2016-07-01)
plugin process crash rate plugin crashes per 1000 use hours "PLUGIN" on 48 Beta jimm 9.53 plugin crashes/khour 12.59 plugin crashes/khour NO, but blassey says plugin crashes do not block e10s release. N/A

Crash rates exclude XP and deprecated OS X operating systems

User Engagement

Responsible rvitillo
Accountable TBD
Supporting measurement/data teams
Consulted Barbara, release management
Informed cpeterson, elan
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Engagement Ratio (DAU/MAU) Engagement Ratio graphs (bug 1251259) rvitillo ~0.22 ~0.35 Yes rvitillo, bsmedberg (2016-05-27)

Jank

Responsible chutten
Accountable bsmedberg
Supporting data team, RyanVM, rvitillo, avih, Softvision
Consulted jgriffiths, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
GC pauses GC_MAX_PAUSE_MS (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Cycle Collector pauses CYCLE_COLLECTOR_MAX_PAUSE (parent, children) beta 46 experiment rvitillo YES: e10s is slightly better chutten and jgriffiths (2016-04-08)
Event loop lag INPUT_EVENT_RESPONSE_MS beta 46 experiment

Nightly 48 telemetry

chutten e10s is different but not worse chutten and jgriffiths (2016-04-08)
Browser responsiveness during page load Talos tp5o_responsiveness Perfherder YES: e10s is 39%–88% faster chutten and jgriffiths (2016-04-08)

Page Load

Responsible jimm
Accountable RyanVM
Supporting jmaher
Consulted jgriffiths, rvitillo
Informed cpeterson, elan, release management
Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o summary ±5% Perfherder 248.32 (Linux)

230.35 (Win8)

246.28 (Linux)

213.51 (Win8)

YES: -0.82% to -7.31% improvement jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

FX_PAGE_LOAD_MS ±5% beta 46 experiment (bug 1228020) rvitillo e10s is much better. YES: e10s is much better. jgriffiths, RyanVM (2016-04-01)

jimm (2016-04-04)

Startup/Shutdown Time

Responsible rvitillo
Accountable
Supporting
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Release Criteria Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Talos tests ts_paint ±10% Perfherder 863 (Win7) 433 (Win7) YES: 43–55% better (bug 1174767) jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Time to first paint SIMPLE_MEASURES_FIRSTPAINT ±10%

("simpleMeasurements/firstPaint")

beta 46 experiment rvitillo 4468 ms 4391 ms YES: e10s is slightly better jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

Shutdown time SIMPLE_MEASURES_SHUTDOWNDURATION ±25%

("simpleMeasurements/shutdownDuration")

beta 46 experiment rvitillo 1156 ms 1190 ms YES: e10s is slightly worse, but within ±25% tolerance. jgriffiths, rvitillo (2016-04-01)

jimm (2016-04-04)

  • In telemetry performance reports, search for "simpleMeasurements/firstPaint", "impleMeasurements/sessionRestored", and "simpleMeasurements/shutdownDuration".

Scrolling

Responsible kats
Accountable RyanVM
Supporting avih, jmaher, graphics team (mstange)
Consulted jgriffiths, jmaher
Informed cpeterson, elan, release management
Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
FX_REFRESH_DRIVER_SYNC_SCROLL_FRAME_DELAY_MS beta 46 experiment avih YES: e10s is faster kats and jgriffiths (2016-04-08)
tp5o_scroll Perfherder kats YES: e10s is 6% – 53% better kats and jgriffiths (2016-04-08)
tscrollx Perfherder kats YES: e10s is 7% – 58% better kats and jgriffiths (2016-04-08)

Plugin Jank

Responsible chutten
Accountable aklotz
Supporting bsmedberg
Consulted Barbara
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By Criteria Met? Criteria Met Signed-off By
Plugin module initialization jank time BLOCKED_ON_PLUGIN_MODULE_INIT_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is better.

Java: YES. e10s is better.

Silverlight: YES. e10s is better.

Unity: YES. e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin initialization jank time BLOCKED_ON_PLUGIN_INSTANCE_INIT_MS ±5% Beta 46 experiment chutten Flash: MAYBE? e10s is slightly worse.

Java: YES. e10s is slightly better.

Silverlight: MAYBE? e10s is slightly worse.

Unity: YES! e10s is better.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Plugin destruction jank time BLOCKED_ON_PLUGIN_INSTANCE_DESTROY_MS ±5% Beta 46 experiment chutten Flash: YES. e10s is different, but not worse.

Java: MAYBE? e10s is slightly worse.

Silverlight: MAYBE? improvement on extreme lag (>2 seconds); regression on moderate lag (200–700ms)

Unity: YES. e10s is different, but not worse.

chutten (2016-04-08, bug 1261362)

aklotz (2016-04-09, bug 1261362) Barbara (2016-04-18)

Memory Usage

Responsible erahm
Accountable blassey
Supporting jmaher (Talos)
Consulted Barbara
Informed cpeterson, elan, release management
Metric Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
tp5o Main_RSS Perfherder 401623141.33 (OS X) 331418381.64 (OS X) e10s is 17%–35% better erahm (2016-04-15), Barbara (2016-04-18)
MEMORY_TOTAL ±25% (browser + content process) beta 46 experiment and erahm's blog post e10s is 10–20% worse, but within our ±25% tolerance erahm (2016-04-15), Barbara (2016-04-18)
VSIZE_MAX_CONTIGUOUS (browser + content process) beta 46 experiment e10s is slightly different, but not worse? erahm (2016-04-15), Barbara (2016-04-18)

UI Smoothness

Responsible mconley
Accountable RyanVM
Supporting jmaher, avih, gabor
Consulted Barbara
Informed cpeterson, elan, release management

Performance numbers below are broken down for both e10s and enabling APZ. Negative numbers represent performance gains.

Metric Metric Owner Analysis Measurements Bugs Criteria Met? Criteria Met Signed-off By
TART ±5% (Tab Animation Regression Test) mconley Perfherder

e10s / APZ
xp = 4.4% combined
win7 = 1.8% / 9%
win8 = -0.6% / 10% (improvement)
osx = 6% / 3.5%
linux = 0.5% / 16%

e10s: bug 1251699
apz:bug 1253412

Borderline mconley (2016-04-19), Barbara (2016-04-20), avih (2016-04-21)
tpaint ±5% (first paint for a new window) mconley Perfherder

xp = 0%
win7 = 9.3%
win8 = 3.4%
osx = -11% (improvement)
linux = 7%

bug 1174770 Borderline mconley (2016-04-19), avih (2016-04-21), Barbara (2016-04-22)
tresize ±5% (window resize responsiveness) jimm Perfherder

e10s / APZ
xp = 1.0% combined
win7 = 0.0%
win8 = -22% (improvement overall)
osx = 1.0% combined
linux = 7% / 2.0%

bug 1252947 Yes jgriffiths (2016-03-17, bug 1252947#c23), mconley (2016-04-21)
tps ±5% (tab switch speed) mconley Perfherder

xp = -27%
win7 = -28%
win8 = -34%
osx = -30%
linux = -19%

e10s: bug 1186585
apz: bug 1216924

Yes mconley (2016-04-19), Barbara (2016-04-20)
FX_TAB_SWITCH_TOTAL_MS Gabor beta 46 experiment 52.66 ms (median) < 82.61 ms (median) bug 1252031 Yes gabor (2016-03-23, bug 1252031#c2), Barbara (2016-04-21)
CART ±5% (Customize Animation Regression Test) mconley Perfherder

e10s / APZ
xp = improvement
win7 = 4.0% / 2.5%
win8 = 2.0% / 2.5%
osx = 3.7% / 2.7%
linux = 6.8% / 6.8%

e10s: bug 1252768
apz: bug 1253380

Borderline avih (2016-04-21), mconley (2016-04-19), Barbara (2016-04-22)

Graphics Performance

Responsible Milan
Accountable RyanVM
Supporting gw280
Consulted Barbara, jmaher, graphics team
Informed cpeterson, elan, release management
Metric Metric Owner Analysis non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
glterrain ±5% jgilbert Perfherder 18 19 Mixed results from 3% worse to 66% better Barbara (2016-05-06), Milan (2016-05-06)
tcanvasmark ±5% Perfherder 6444 6301 Mixed results from 2% worse to 3% better Barbara (2016-05-06), Milan (2016-05-06)

Slow Scripts

Responsible azhang
Accountable bsmedberg
Supporting
Consulted Barbara, billm
Informed cpeterson, elan, release management
Description Metric Analysis Analysis By non-e10s value e10s value Criteria Met? Criteria Met Signed-off By
Slow script warnings SLOW_SCRIPT_PAGE_COUNT beta 47 experiment azhang 0.00117087669174 warnings per use hour (aka 854 hours/warning) 0.00158655016424 warnings per use hour (aka 630 hours/warning)

(bug 1260769)

Barbara and Jim are OK with this regression because display rates are extremely low in both modes. Barbara, jimm, billm (2016-05-25)

Tests

All tests disabled for e10s must be triaged and:

  • reenabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s?
  • or deleted?

We want each test directory to be signed-off by the feature area's owner.

Responsible Blake, Felipe
Accountable jimm
Supporting dolske (coordinating people fixing front-end tests)
Consulted jgriffiths
Informed cpeterson, elan, release management
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan RyanVM In progress: RyanVM says he is waiting for feedback from blassey, jimm, and milan. TODO
Execute test plan RyanVM In progress TODO
Re-enable disabled e10s tests Felipe, Blake In progress TODO
  • Notes from jimm: This looks good. I've removed XP from the operating system lists.

Release Blocking Bugs

All M9 bugs must be fixed before we release.

Responsible jimm
Accountable blassey
Supporting release management
Consulted jgriffiths
Informed cpeterson, elan

M9 bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Release Criteria bugs

Full Query
ID Summary Whiteboard Keywords Assigned to
1063169 [meta] tracking performance issues with e10s [webvr] meta
1250350 Talos svg results for e10s seem invalid
1260769 [e10s] SLOW_SCRIPT_PAGE_COUNT regressed under e10s regression

3 Total; 3 Open (100%); 0 Resolved (0%); 0 Verified (0%);


APZ Bugs

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);


Accessibility

  • Accessibility (a11y) does not block the initial release of e10s because e10s will only be enabled for users not using a11y.

Add-ons

  • Add-on support does not block the initial release of e10s because e10s will only be enabled for users without add-ons.
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Release_Criteria&oldid=1139838"