Firefox 38 Post mortem

• (sylvestre) major release out of the schedule should be avoided at all cost
  • We should not do that ever again +1
  • Too much confusion
  • Workflows not planned for this
  • Add a lot of stress to a complex situation
  • The branch mechanics are not planned for this
  • The tracking flags don't manage this complexity
  • affects quality of other releases (not getting enough attention) +1

• (sylvestre) Building beta from m-r
  • release notes broke
  • The merge failed (we had the release branding)
    • and we almost produced RC with beta branding
  • Releng tools had to be updated
  • Didn't get data for android beta 8
  • we have to do some l10n updates
  • RelMan (who starts builds) not aware of status of reverting beta from m-r work around - need coordination
  • Issues described here: http://release.mozilla.org/statistics/38/2015/04/29/fx-38-b6-to-b8.html

• (KaiRo) Doing a build2 of b8 resulted in long wait times for CDN purging, on top of being late already
  • Bug 1159226 filed for (auto-)starting purging process when starting a build2 - solution might be to avoid purging completely, though

• Pocket
  • Last minute
  • Not a clear plan
  • Hacky solution costs additional time
  • Focus on feature rather than quality (performance)
  • Community taken by surprise on integrating proprietary service +1 why not an addon? Man, this is like half of the feedback.

• Suggested tiles
  • The QA was planned way too late (beta 8)
  • Postponed to 39
  • We were informed of the testing on the last minute (during the internal meeting)
  • Not a good coordination between relman and the content services team
  • still active development on this feature during beta (i.e. now)

• Need more QA focused on graphics
  • (sylvestre) We are starting to work with beta breakers to get better gfx testing
  • (lmandel) second release in a row that was broken by a typo in graphics blocklist entries

• (lmandel) We tried to rush 38.0rc build 3 out and broke updates. This is both a failure of communication in relman and a failure of balrog.
  • should be blocked on automation sides (and manual double checks in meantime)

• (lmandel) relnotes weren't published correctly. Were redirecting to the beta notes for some reason. I have also seen that relnotes are not updated on stage sometimes (as with the 38.0.5b1 notes).

• (lmandel) sec advisories weren't live at the same time as release. Couldn't find abillings online. Should relman be responsible or have the ability to publish these notices?
  • (sylvestre) Or move the sec adv to nucleus (where are they hosted now?)

• (lmandel) We still have work to do with the comms/pr teams. Are we posting to the Mozilla blog for each release? How do we handle pr related communication at a time other than the morning of the release?

• (KaiRo) Socorro issues right on release Monday/Tuesday were troublesome - there was no code change in Socorro itself in the days before though, so might be bad coincidence. The cause was mostly found but not exactly sure why it came up that Monday specifically.

• (KaiRo) getting second RC out without KaiRo present - was good to see that we can do that, thanks to everyone who did work to get it done.
  • (sylvestre) You can now take holidays ;)

• (KaiRo) AMD crash in second RC in function not covered by dmajor's patch
  • increase of the AMD cpu crash during the 38 cycle

• (lmandel) firstrun is hardcoded in product. Required a rebuild for ESR. (will this be true for future ESRs as well?)
  • we disabled Hello (and EME) in 38 ESR but the "what's new" was mentioning Hello.
  • Normally we expect ESR to have the same features as mainline Firefox

• (lmandel) Separate fixes in Firefox and ESR 38.0.1 but no way to publish separate relnotes for these two releases as Firefox and ESR use the same URL structure.

• (sylvestre) what should we do with these post mortem notes?
  • (lmandel) publish summary to wiki, file bugs to get issues addressed
  • review and summarize (and publish) post mortems with each ESR, maybe

• Android crash checked in as part of b10 was primary driver for 38.0.1
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1159262
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1163841

• (KaiRo) startup crash we shipped 38.0.1 for was actually known throughout the cycle and even after failed fix
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1154703