Notes for Firefox 71 release post mortem

10am PT January 7, 2020 Zoom Meeting ID: 266 532 573 slack: #release-coordination

Attendees

pascal, RyanVM, telin, tmaity, astevenson, philipp, marcia, lizzard,mconley, jcristau

Nightly: Sept 02 to Oct 21 (7 weeks)
Beta: Oct 21 to Dec 2 (6 weeks)
Uplifts to Beta: 223 (353 in 70, a 7 week cycle)
Uplifts to RC: 11 (17 in 70)
Fixed in 71: patches from 4090 bugs (5367 in 70)
- of which 721 regressions
We had 22 desktop features (as per QA Status Report), 1 bug work, 4 off train features
- 11 features moved to future releases/backlog
PI Request deadline was Sep 13, there were a few Search related PI requests submitted post the deadline however, those got cancelled and moved for future releases (PI -253, 252)
- Celebration Toast went through exception approval, "DevTools What's New Panel" feature request was submitted pretty late (a week prior to prelim status report), went through exception approval process
Technical documentation deadline was Sep 27, 2 features missed the deadline
Feature Complete milestone was Oct 11, apart from ETP 1 feature missed the deadline

Top crashers were identified in nightly and fixed or patches backed out. We entered the beta cycle in a stable state and focused on feature uplifts only
Stability confirmed to be good throughout the beta cycle and after the release
No dot release (didn't happen since 42 in 2015):yay:
No surprises in Features, we had a few carryovers from 70 a couple of features slipped to 72 but no surprise feature got in.
I approved the activation of PIP for Windows in the release channel during the beta cycle as the quality was very good, this paid off with good press and users feedback with no regressions reported
No WNP problems, no issue with anything marketing/PR related or off train
DNS over HTTPS offtrain deployment didn't cause relman issues
[QA] Collaboration with the engineering team responsible for Fission improved significantly.

4 builds for RC, 3 RCs shipped, one just before the release for a potential crash in DHH. More RCs than we like, some of the RCs were "better safe than sorry" ones but it paid off.
RC week was thanksgiving week in the US, lots of people were away so response time was low (releng and devs) on uplift, it added some extra stress but caused no real probleml in the end
A Ubuntu top crash (https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1850529) for 3rd party builds caused by a Clang toolchain bug was worrying during the whole beta cycle, we contacted the Ubuntu maintainer and they shipped non-crashy RC and final builds to their users
LSNG was not ready for release and we deactivated it in beta 10 (in agreement with the dev team) so as to give the dev team feedback on beta population volume and still have 2 weeks with the pre-LDNG path exposure -> Should we take a final decision about LSNG? It has been slipping release to release since at least 66
- Also disabled in 72 and intending to disable in 73 before shipping. RyanVM to follow-up with janv for 73.
We had a deployment bug for security issues on release day caused by a YAML formatting bug not caught by our linters (https://github.com/mozilla/foundation-security-advisories/issues/37) -> We found the source cause and updated our linter to catch it next time.

[QA] Communication with the engineering team responsible for DNS over HTTPS (PI-357) was for the most part poor, resulting in a lot of wasted time.
- shipping method changed with no prior notification
- testing requirements changed with short notice or without notice
- significantly delayed replies. Comments from Engineering with respect to issues we couldn’t work around wasn't helpful
Over 100 man-hours spent on a 70.0.2 dot release (Close the FxA MAU Gap) that eventually never happened. Fx71 commitments were affected as a result.
- Includes : attending meetings, reading documentations, creating clarifications doc, Test Plan/Test Case creation and time spent on Test execution, Reporting.

[Read-Only] from QA

[FxR] Undocumented changes and rushed development (many regressions being introduced). QA resources are not balanced well for the large number of testing requests. (Understaffed)
[Add-ons QA] Usage of qe-verify+/- flags still needs to improve. Due to the fact that Engineering teams don’t use those enough, QA invests sometimes unnecessary time in bugs which don’t need testing (or can’t be verified manually)
[Ecosystem QA] Normandy team – still makes changes and deploys without letting QA know about those.