Notes for 64.0 release post mortem

10am PT Tuesday January 15 (after the channel meeting) Vidyo Channel: ReleaseCoordination IRC: #release-drivers

Some stats

• 7 weeks on nightly (Sept 4 to Oct 22), 7 weeks on beta (Oct 22 to Dec 11)
• milestones for reference: https://docs.google.com/document/d/1MQa4J9OSTc1i60sFcVNyoFNLK8TLBXJlL9W12VoEUAc/edit
• 245 uplifts to beta, 13 uplifts to release
  • slightly lower than 63, which was already on the low side
• uplifts during beta: https://bugzilla.mozilla.org/page.cgi?id=release_tracking_report.html&is_custom=1&q=approval-mozilla-beta%3A%2B%3A20181022-20181203%3A0%3Aand%3A838%2B%3A
• uplifts during rc: https://bugzilla.mozilla.org/page.cgi?id=release_tracking_report.html&is_custom=1&q=approval-mozilla-release%3A%2B%3A20181130-20181209%3A0%3Aand%3A838%2B%3A
• We started Fx64 with 19 desktop features, 8 of them had been moved to future releases (Ref: 64 QA status report)
• We received 2 PI Requests after the deadline (Aug 29) both were small features
• 127 bugs were created in Nightly 64, 31 have been fixed before Release (https://docs.google.com/spreadsheets/d/1IWUx-8AOADdzuKCRl7jo5oQ8djdS0kCjfqkAVAI8jxU/edit#gid=0)
• Features readiness date was Sep 12, 7 features were not ready/code not landed as of Sep 19. 5 Features were still waiting for kick-off meeting

What went well

• no dot release driver on desktop

• first release using shipit v2 (after 2 cycles on beta)

• The Firefox 64 Release Process document was, as with the previous releases, very useful– it helped us ensure optimal coverage for each of the builds that were thrown at us during release-candidates week

• For most of the features, the communication between QA & Eng went well

Challenges

• 1508494 WNP setup
  • should have strings locked down this week for translation

(This has come a bit late from marketing to the l10n team) We may need to give more time to l10n team especially considering holidays https://bugzilla.mozilla.org/show_bug.cgi?id=1510065 has more details.

• • because WNP came in late and didn't go through the normal l10n process there was a bunch of special casing which caused issues both during rc week and again for 64.0.2

• fennec crashes
  • 64.0b15 fixed an issue that was hiding crashes, so we ended up with a bunch of last minute top crashers
  • lack of 64 rc build was a mistake in hindsight
  • blockers were fixed in 64.0.1
  • could build in more frequent checks to play store anr/crash reports into the cycle

• avoidable rc uplifts
  • webcompat: Event.returnValue https://bugzilla.mozilla.org/show_bug.cgi?id=1510985
    • this might happen again: https://bugzilla.mozilla.org/show_bug.cgi?id=1514940
  • webcompat: js error messages https://bugzilla.mozilla.org/show_bug.cgi?id=1498257
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1498257#c11
  • file handling: https://bugzilla.mozilla.org/show_bug.cgi?id=1512496
  • needinfos falling through the cracks and/or incorrect tracking flags (If you're having trouble getting eng to respond ping telin on slack. I'll chase folks down)
    • Someone said kill all-hands! -1
    • Or cancel all holidays -1

• from 64.0.2:
  • incomplete/incorrect translations for major locales (ja https://bugzilla.mozilla.org/1513259, en-CA)
    • should we be worried about this biting us going forward?
  • youtube regression, not reported pre-release?
    • https://bugzilla.mozilla.org/show_bug.cgi?id=1513511

QA updates:

• Reaching out to people to fix smaller issues (WNP or Snap related) was difficult with the All Hands happening the same week.

- Each issue delayed affected tasks by approximately 1 day.

• • The addition of a third release candidate build impacted the team, as we had to sign off a total of 6 builds in a single week.

- 3 x 64.0 Release Candidates, 2 x 65.0 DevEdition, 1 x 60.4 ESR

• • DevEd sign offs are lowest priority, then ESR builds, RC build sign offs to be treated as highest priority

• • We should avoid releasing builds during All Hands, any thoughts?
  • Planning a launch in Dec is always tricky.

• • FastBlock feature: Team faced challenges in terms of confusions with documentation updates and answer to the QA queries were not clear. There were lots of back of forth between QA and Eng team.

• • Devtools Flexbox Inspector - Due to large number of changes made to initial functionality (throughout the features life cycle), which was translated in a TCs redesign for a few times - bringing some extra unplanned workload for the team - Tania to add more details for this one.