Electrolysis/Multi Release Criteria

From MozillaWiki
Jump to: navigation, search

Page to collect and track Electrolysis Multi Process release criteria.

Rollout Criteria

We plan to ship e10s-multi in Firefox 54 / 55 using a staged rollout. See elan's e10s-multi release plan.

Report / Dashboards

Rollout Status

  • Nightly: 4 processes
  • Beta: experiments running, user default is 1 content process
  • Release: 1 content process

Notes

1) We can't rely on Talos numbers generated in automation for beta builds. The default there is one content process. We'll have to use try runs of the beta code base to generate comparisons.

AWSY Notes

1) Maximum number of tabs set by automation is currently 30.

2) test source

Release Criteria

Stability

Reporting Jim Mathies
Data Wrangler
Regression(s) Blake Kaplan, Gabor Krizsanits
Metric Description Report Links Analysis By Criteria Met? Criteria Met Signed-off By
Crash Rate (chrome crashes + (content process(es) - content process(es) shutdown termination crash reports)) / 1000 use hours beta 54 (redash) jimm SO FAR - multi1 / multi4 rates track TBD

Jank

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
GC_MAX_PAUSE_MS Longest GC slice in a single GC cycle in milliseconds, broken down by parent and child process(es). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
CYCLE_COLLECTOR_MAX_PAUSE Longest pause for an individual slice of one cycle collection, including preparation in milliseconds, broken down by parent and child process(es). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
INPUT_EVENT_RESPONSE_MS Time in milliseconds for input event lifetime (created -> fully handled) including traversal to a child process and back. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_TOTAL_MS Time in milliseconds a tab switch takes, including first paint of the tab. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
Metric Description Report Links Analysis By Results for Child Process Cohorts Criteria Met? Criteria Met Signed-off By
1 4
tp5o_responsiveness graph TP5O Talos numbers, comparing current single process beta builds with try runs using the beta code base and four content processes.
Targets: 64-bit opt/pgo builds 		Comparison 20170522

Comparison 20170510

tracy

Win7 32bit: 6.11
Win8 64bit: 4.93
OSX: n/a
Linux: 4.55

Win7 32bit: 6.25 (+2.39%)
Win8 64bit: 4.86 (-1.40%)
OSX: n/a
Linux: 4.52 (-0.59)

YES - note: bug 1362920 TBD

Page Load

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
FX_PAGE_LOAD_MS Time taken to load a page (ms). This includes all static contents, no dynamic content. Loading of about: pages is not counted. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD

UI Smoothness

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Metric Description Report Links Analysis By Results for Child Process Cohorts Criteria Met? Criteria Met Signed-off By
1 4
TART graph Tab animation regression test (Talos). 20170522


20170510

tracy

Win7 32bit: 7.36
Win8 64bit: 6.26
OSX: 11.18
Linux: 6.31

Win7 32bit: 7.34 (-0.27%)
Win8 64bit: 6.21 (-0.81%)
OSX: 11.35 (+1.54%)
Linux: 6.32 (+0.17%)

YES TBD
TPAINT graph First paint for a new window (Talos). 20170522


20170510

tracy

Win7 32bit: 298.80
Win8 64bit: 283.82
OSX: 323.16
Linux: 277.12

Win7 32bit: 296.06 (-0.92%)
Win8 64bit: 287.60 (+1.33%)
OSX: 322.43 (-0.23%)
Linux: 277.34 (+0.08%)

YES TBD
TRESIZE graph Window resize responsiveness (Talos). 20170522


20170510

tracy

Win7 32bit: 11.99
Win8 64bit: 10.65
OSX: 28.87
Linux: 23.34

Win7 32bit: 11.95 (-0.33%)
Win8 64bit: 10.61 (-0.42%)
OSX: 28.89 (+0.07%)
Linux: 23.90 (+2.38%)

YES TBD

Memory Usage

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
MEMORY_TOTAL Total memory across all processes (KB). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
MEMORY_VSIZE_MAX_CONTIGUOUS Maximum-sized block of contiguous virtual memory (KB). ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD

Are We Slim Yet (AWSY)

RSS chrome process + USS content processes (Talos)

5/10/2017 Perfherder

(insert passing criteria declaration)

Windows 7 Opt 32-bit
Metric Regression Criteria Met? Sign-off
Resident Memory Fresh start -0.47%
Resident Memory Fresh start [+30s] -0.67%
Resident Memory After tabs open 29.80%
Resident Memory After tabs open [+30s] 32.87%
Resident Memory After tabs open [+30s, forced GC] 26.59%
Resident Memory Tabs closed -29.74%
Resident Memory Tabs closed [+30s] -18.40%
Resident Memory Tabs closed [+30s, forced GC] -12.14%
Linux Opt 32-bit
Metric Regression Criteria Met? Sign-off
Resident Memory Fresh start 0.11%
Resident Memory Fresh start [+30s] 0.27%
Resident Memory After tabs open 19.87%
Resident Memory After tabs open [+30s] 18.98%
Resident Memory After tabs open [+30s, forced GC] 13.15%
Resident Memory Tabs closed -20.46%
Resident Memory Tabs closed [+30s] -10.59%
Resident Memory Tabs closed [+30s, forced GC] -7.76%
Linux Opt 64-bit
Metric Regression Criteria Met? Sign-off
Resident Memory Fresh start -1.76%
Resident Memory Fresh start [+30s] -1.80%
Resident Memory After tabs open 22.51%
Resident Memory After tabs open [+30s] 21.92%
Resident Memory After tabs open [+30s, forced GC] 15.06%
Resident Memory Tabs closed -21.38%
Resident Memory Tabs closed [+30s] -10.48%
Resident Memory Tabs closed [+30s, forced GC] -7.23%

Tab Switching

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits, Mike Conley
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
FX_TAB_SWITCH_UPDATE_MS Time in ms spent updating UI in response to a tab switch. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_TOTAL_E10S_MS Time in ms between tab selection and tab content paint. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_SPINNER_VISIBLE_MS Time in ms spent updating UI in response to a tab switch. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_SWITCH_SPINNER_VISIBLE_LONG_MS If the spinner interstitial displays during tab switching, records the time in ms the graphic is visible. This probe is similar to FX_TAB_SWITCH_SPINNER_VISIBLE_MS, but is for truly degenerate cases. ben/tracy

Win: YES
OSX: limited data
Linux: limited data

TBD
FX_TAB_REMOTE_NAVIGATION_DELAY_MS
 Time taken in milliseconds between the browser sending a naviagion event to content and content receiving it. This message can be either SessionStore:restoreTabContent or WebNavigation:LoadURI and these names are used as keys for this histogram. This is e10s only and recorded in the content process. Measures delays associated with child process creation. TELEMETRY LINK ben/tracy

Win: YES
OSX: YES
Linux: YES

TBD
Metric Description Report Links Analysis By Results for Child Process Cohorts Criteria Met? Criteria Met Signed-off By
1 4
TPS Test graph Tab switch timing (Talos) 20170522


20170510

tracy

Win7-32bit: 26.87
Win8-64bit: 24.06
OSX: 29.39
Linux: 29.74

Win7-32bit: 28.80 (+10.92%)
Win8-64bit: 22.83 (-5.11%)
OSX: 25.48 (-13.30%)
Linux: 28.66 (-3.65%)

YES - note: bug 1362920 TBD

Slow Scripts

Reporting Jim Mathies
Data Wrangler Ben Miroglio
Regression(s) Blake Kaplan, Gabor Krizsanits
Report Link e10s-multi performance in Telemetry
Metric Description Analysis By Criteria Met? Criteria Met Signed-off By
SLOW_SCRIPT_NOTICE_COUNT Total count of slow script notices displayed to user. ben/tracy

Win: YES
OSX: limited data
Linux: no data

TBD
SLOW_SCRIPT_PAGE_COUNT The number of pages that trigger slow script notices. ben/tracy

Win: YES
OSX: limited data
Linux: no data

TBD
SLOW_SCRIPT_NOTIFY_DELAY The difference between the js slow script timeout for content set in prefs and the actual time we waited before displaying the notification in milliseconds. ben/tracy

Win: YES
OSX: limited data
Linux: no data

TBD

Tests

Reporting Blake Kaplan / Gabor Krizsanits
Accountable Everyone
Sign Off Module Owners / Jeff Griffiths

All unit tests disabled for e10s-multi must be triaged and:

  • re-enabled (and fixed, if necessary)
  • or annotated that the test is irrelevant for e10s-multi
  • or deleted

We want each test directory to be signed-off by the feature area's owner.

QA Test Plan

Responsible Erin Lancaster
Accountable SoftVision / tracy
Sign Off Jeff Griffiths / tracy
Task Owner Criteria Met? Criteria Met Signed-off By
Write test plan Tracy Done tracy
Execute test plan Tracy, Ben M, SV on going perf testing. so far so good tbd
Retrieved from "https://wiki.mozilla.org/index.php?title=Electrolysis/Multi_Release_Criteria&oldid=1171816"