Firefox/Channels/Postmortem/68

From MozillaWiki
Jump to: navigation, search

Notes for Firefox 68 release post mortem

10am PT August 20, 2019 Zoom Meeting ID: 266 532 573 IRC: #release-drivers / slack: #release-coordination

Attendees

Julien, Philipp, Overholt, Ritu, PLawless, lizzard, RyanVM, dveditz, Tom, Thomas, Yan, mikedeboer,tania

Stats

  • PI Requests received on time (prior to deadline) : 19 (Desktop)
    • Late PI Request: 9, Accepted: 5, Rejected: 4, 1 request went through Exception approval process (Desktop QA)
  • 9 QA requests were cancelled/moved to Future Release to prioritize Trailhead work
  •  % of PI requests (Desktop) with initial documentation received after deadline : 19.05% (slightly better than 67)
  •  % of Desktop Features with Late Code Delivery: 31.82% (slightly better than 67)
    • This means features that landed after FC
  • Total number of bugs found by QA:
    • Nightly :75
    • Beta :48
    • some testing was canceled to prioritize trailhead testing and dot releases

New release process

  • no mandatory testing in nightly
  • exception process for missed deadlines or feature uplifts
    • [julien] I heard some concern about the time it takes to get an exception approved adding to an already-delayed feature
    • most features asked for Nightly testing
    • [julien] I have an action item to make the exception process doc more discoverable from release process docs
      • link to it from the cross functional
      • We should also make the exception requests sit in the same folder
    • https://docs.google.com/document/d/1jOTngDV_WWJGelTz4205ux4i5sAWfYNILHYmLLmHYeQ/edit -> Exception request document
    • Relman team to figure out how to educate more feature owners on how and when to file an exception request
    • List of exceptionr requests for 68 -
    • AI: jcristau to find the number of exception requests we got (e.g. for late code delivery)


What went well

  • fennec on esr repo
  • Moving to JIRA seems successful as there was no email request received for Fx68 (desktop qa) requests.
  • Response from Eng team in reviewing QA artefacts was quick, feedback received on time for most of the features, apart from Desktop this is applicable for Add-ons QA and Mobile QA
  • Introduction of the Technical documentation guideline seems to be helpful as in some cases we were able to identify design gaps and report them beforehand.


Challenges

  • about:addons
    • much confusion about the status of the feature going into beta
  • Priority” values of the PI tickets - Most (or all?) of the PI requests were of medium priority caused more work while prioritizing them during Trailhead
    • involving product management in prioritization could help
  • majority of the features went through Nightly Testing
    • is it slightly weird that above we're talking about the number of uplifts to beta being high but we're trying to only test in beta? Yes
    • number of beta uplifts wasn't unusually high
    • we clarified that we aren't necessarily trying to reduce Nightly testing
  • Still difficult to plan resources each cycle for new features based on the data initially available for them/ minimal information added to PI ticket.
  • Communication with Engineering didn't go well on Installer related features, scope changes and unclear answers to the clarifications resulted some misunderstandings.
  • Mobile QA - Three features were moved to next Fx release due to the fact that the implementation wasn't done on time and QA didn't have enough time to test.
  • Add-Ons QA - Patches have been uplifted late, causing the testing process to fall behind the planned timeline required for extensive testing.
  • For the Quantum bar, although there was a reasonable amount of documentation, for the address bar in general, search, telemetry, db structure, etc. that documentation is lacking: even on the info that exists, you need the right person to point you to the right place.
  • LSNG
    • stability issues in the beginning of beta were fixed
    • disabled in rc2 for perf concern (Bug 1562942)
      • Do we need better Go/NoGo criteria for this feature?
    • do we have telemetry for that? could the issue have been caught earlier?
      • we are working on automated tests for "existing profiles" and "profiles with addons installed" so it'll be better in the future
  • QuantumBar
    • a number of last minute uplifts (https://mzl.la/2KI6zG1)
      • [M DeBoer] Late uplifts in RC were driven by user feedback and not something easily identifable early on.
      • [M DeBoer] Documentation concerns stem from the age and complexity of this project. Will ensure that Bridget and/or Michaels contact info are obvious and attached to all reporting sources and requests. PI request, Trello, tech docs, etc.
    • should we have said no?
    • should it have been deferred?
      • the late uplifts weren't necessarily blockers, but they broke some power users' workflows; we don't know how many
    • bridget and mikedeboer are points of contact on search
  • WebRender rollout
    • confusion right before launch (bug 1562966)
    • should features that plan a gradual rollout on release get the same treatment on beta, so we make sure prefs are in the expected state and we can beta-test the rollout itself?
  • localfiles-sec-july19
    • very old bug (Bug 1558299)
    • incident over July 4th weekend
    • ended up fixing in 68 rc3
    • needed a regression fix in 68.0.2
  • new ESR branch
    • delayed build1 gtb
    • patches to disable features not ready in time
    • tooling issues (shipit)
Retrieved from "https://wiki.mozilla.org/index.php?title=Firefox/Channels/Postmortem/68&oldid=1216887"