Firefox 37 Post mortem

• (lmandel) We didn't check that all required fixes were in for ESR 31.6.0 and had to spin a second build

• (lmandel) We didn't stabilize early enough and had to take fixes and release with data obtained over the weekend.
• (kairo) We also decided to release even though we knew we were not meeting our usual stability level, and release stats show we are regressing release further due to that: http://arewestableyet.com/graph/?fxrel
• (lmandel) Graphics related crashes continue to bite us in release. (Required 37.0.1.) We need a better way to identify these issues earlier.
  • (florinmezei) Don't we still have a Graphics QA team testing on various hardware configurations?
• (lmandel) We had a security issue that needed to be patched and may have warranted a point release on it's own. Not sure if we can do anything better about this.
• (bhearsum) Ownership of non-standard update testing (Eg, whatsnew pages, first run, partner repacks)
• (lmandel) There were a number of questions about why Firefox showed no update to 37.0. We shut off updates completely in this version. We should probably communicate when we do this as it is a surprise.
• (lmandel) We still have no mitigation for start-up crashes other than a reinstall.
  • (kairo) We also probably have lost quite a few of the people from beta that have seen startup crashes regularly, mostly because we were slow to address gfx startup crashes in beta during the OMTC landing process.
• (lmandel) We were able to release 37.0.1 on Easter Friday when many people were off and we had somewhat limited coverage.
  • (florinmezei) Releasing on Fridays is bad practice... releasing on holiday/almost-holiday Fridays is even worse. Not saying that we could have done anything to avoid it in this case, but I think in general we should avoid doing that as much as possible, for lots of obvious reasons.
• (lmandel) We were unable to pref off MSE on OSX until late in the cycle due to waiting on our partners.
• (lmandel) EME was supposed to be completely isolated for 37 testing turned out not to be. We accepted risk of code change (that did result in breakage) without any benefit because our partners were not ready.
• (lmandel) Still lots of pressure to push features late in the cycle. Quality seems to be taking a hit.
  • (florinmezei) I think we should make it a habit of having new features ride all 3 trains (Nightly, Aurora, Beta) after they are mostly dev complete, and in testable form. That would mean no major UI/UX changes, refactoring, redesign in Beta phase.
  • (florinmezei) Note the not so good situation of some of the Firefox 38 features:
    • Reading List - not really ready for testing until pretty much before 38 landed in Beta - massive uplifts in 38 Beta 2 (high risk)
    • Suggested Tiles - pretty much hidden until before 38 landed in Beta
• (lmandel) Some tracked bugs are not moving. We need to do a better job of highlighting the issues that are important for a release.