Firefox 55 Post-mortem

Updates still disabled (all platforms except Android) About half of the population is on 55

Reminder: dot releases have a HUGE impact on retention. We have data which shows a drop of 20% in previous releases because of that (or the bugs that we fixed)

• dot releases
  • Analysis https://docs.google.com/spreadsheets/d/1_B-HU-ZgR3IPaPvtgXQl0R3MjxOH0ehDJde8U6PG0II/edit#gid=0
  • Because of the impacts of some bugs, we had to disable updates, take patches, release asap.
  • Different severe drivers
    • Pages not longer showing for pks libs - Was fixed on nightly - Reported several months ago
    • Tabs are all restored as blank frequently - Known this mid may
    • White, unreadable text on Flash content - Known in mid March
    • YouTube thumbnail upload issue - Reported a month ago
    • etc
  • None of the drivers tagged

• Still working on 55.0.4 - Potential drivers:
  • Bug 1388611 - data lost
  • Slow flash (1394105)
  • Too many requests with WebSocket - https://bugzilla.mozilla.org/show_bug.cgi?id=1394564

• Potential reasons:
  • Everybody is focusing on 57
  • 55 was on nightly for two cycles (no previous cases of that)
  • PTOs
  • 6 to 8 weeks of stabilization removed

• Potential mitigation strategies?
  • Accept that we will have much more dot releases and plan for it
    • Need more QE outside of eastern Europe
  • Having more people to do the upfront triage, contractor ?
  • Communication from leadership (dcamp?) with relay from management to explain the different expectations
  • More involvement from eng team in triage
  • More ownership of eng leads

• Update super long. Still half of the release population on 54
  • security issues
  • Proposition

[kohei] In the future, is it possible to separate a major update from a security update to mitigate security risks? For example, put all security fixes in 55.1.0 shipping in the midst of the release cycle after dealing with the dot releases, rather than delivering all of them with 55.0.0. We are just lucky that we don't have any 0-day attack yet while the auto update has been disabled almost for 3 weeks. The security bugs are still private, but the advisories were published at the same time of the major release, and the patches can be found easily from the code repository.

• 55 staged rollout
  • Too fast? Too slow?
  • We will share a plan for 57 (using devedition population earlier 57b)

brute force this issue eng leadership to be more engage

• last minute fennec changes to EOL API 15 and disable one touch pinch gesture
  • why was this not caught earlier?
    • EOL API15 has been on the radar. last minute joe saw a spike in downloads. given that we were tight on resourcing working on higher priority features, decision was made to wait till downloads started decreasing again. nonetheless, afaik it was always targeting 56
    • on june 13 jcheng commented on the trello card: "Our ICS population has been trending downwards for the past year until 2 weeks ago. Comparing June 11 to May 30, ICS (Android 4.0) installs on active devices increased by 12.4% (based on Google Play data). We need to figure out why this is happening before making a call here. Looking into some data and will get back."
  • thank you for bringing up the one touch pinch as there is room for improvement on this process. this was landed by a contributor and was r+. doesnt look like there was any official QA till a month after landing. this issue was not surfaced till andreas was playing around with it and found that it was janky.
  • EOL API - was this driven by build problems? Audio features provided by Android which were hard for the media team to maintain, they pushed hard to disable this version of Android
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1316462 - Removing the API
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1379190 - Remove AndroidDecoderReader - "The AndroidDecoderReader is no longer maintained and is dragging the rest of the playback architecture back."

[marcia] Lots of the Google play comments for release mention that we are missing the SD card option - does know anyone anything about that?

• What's new page keeps causing headaches, this time due to breakage in firefox

• Update testing

• Balrog sign off
  • Rubber stamping of the rules from RM
  • QE signs off twice (emails + balrog)
  • Disabling updates should not require 3 different teams