Releases:Release Post Mortem:2016-04-06
From MozillaWiki
Meeting Details
- 3:30pm ET
- Vidyo - Release Engineering room
- Dial-in:
- 650-903-0800 or 650-215-1282 x92 Conf# 98225 (US/INTL)
- 1-800-707-2533 (pin 369) Conf# 98225 (US)
- #releaseduty on irc.mozilla.org
- https://trello.com/b/MXHaVRcP/release-promotion-meeting
« previous week |
index |
next week »
< most recent |
upcoming >
Release Duty
- FF 46 cycle: kmoir
Misc
Not Shipped
Fennec 46.0b6 (not shipped due to infra issue)
Firefox 46.0b6 (not shipped due to infra issue)
- Commit: https://hg.mozilla.org/releases/mozilla-beta/rev/97b81104ac035d6a8f6ed59a1aad63fcc23e73c8
- Task graph: https://tools.taskcluster.net/task-graph-inspector/#dpjEFnSWTz20BnFhY6OD-g/
Thunderbird 38.7.1 (kmoir/mtabara) (not shipped due to infra issue)
- pushed live with 25% update rate as per tbdrivers email instruction
- http://hg.mozilla.org/releases/comm-esr38/rev/537d63e899c7
Issues
- repack_2/10 failed for Thunderbird 38.7.1 build1 on macosx64
- repack_5/10 failed for Thunderbird 38.7.1 build1 on macosx64
- repack_6/10 failed for Thunderbird 38.7.1 build1 on macosx64
- repack_9/10 failed for Thunderbird 38.7.1 build1 on macosx64
- Thunderbird 38.7.1 SHA512SUMS file is incorrect for one l10n chunk on mac bug 1260986
Shipped
Thunderbird 38.7.2 (nthomas/kmoir)
- https://hg.mozilla.org/releases/comm-esr38/537d63e899c7
- repack_1/10 failed for Thunderbird 38.7.2 build1 on win32. Looks like a malformed build machine, it couldn't do any uploads. Same machine as win32 repack 4/10. Rerun by nthomas.
- Ready to ship message to the release channelwhich normally comes at the same time as
completed release_final_verification did not arrive until late ET Monday due to waiting for antivirus job to complete.
Thunderbird 45.0b4-build1 (nthomas/kmoir)
- https://hg.mozilla.org/releases/comm-beta/983d1ccaea8f
- First submission of build1 failure due to problem with l10n milestone that release runner caught. This was fixed and submitted to ship it again as build1
- linux and linux 64 builds failed with errors similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1261202 but c7:60:58:e5:d0:3f:1c:71:88:f8:e4:1b:04:05:28:0a:fb:1f:0f:40, builds restarted automatically
- thunderbird source failed with issue and was restarted automatically
- update verify 1, 2, 3 for linux and linux64 failed due to missing gtk3 libraries, the same issue as the last beta
Retrieving 'http://archive.mozilla.org/pub/mozilla.org/thunderbird/candidates/45.0b4-candidates/build1/linux-x86_64/de/thunderbird-45.0b4.tar.bz2' from cache... ../../update/updater: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory
Firefox 46.0b7
- http://hg.mozilla.org/releases/mozilla-beta/191f5eb4cbd72590277296cdb90d355adb347d45
- build1 - The build was started before win opt builds were ready. As a result release runner failed and marked the release as failed.
- beetmover credentials issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1260367#c5
- had to restart a l10n repack via tctalker https://tools.taskcluster.net/task-inspector/#Df9cyZe-SIqqNJDbpLAGZA/1. Previous job hung and hit the timeout.
- many repacks are failing due to issues with capacity on our signing servers bug 1261498 and bug 1261498
- bug 1261637 - Balrog serves 46.0b5 partials in 46.0b7 on beta-localtest
- nthomas created a stub for to add Firefox-46.0b5-build2 back to balrog so update tests would pass. We reran the jobs via tctalker and advised QE they could start running update tests.
- version bump failed because hg.m.o cert changed since the build was started. The build amis needed to be regenerated with the new certs in 1261508 bug 1261508
Fennec 46.0b7
- http://hg.mozilla.org/releases/mozilla-beta/fb03a1c0af86b816918e21a5b10db353b6be339b
- bug 1261202 - hg clone failed
Abort: certificate for s3-external-1.amazonaws.com has unexpected fingerprint c7:60:58:e5:d0:3f:1c:71:88:f8:e4:1b:04:05:28:0a:fb:1f:0f:40
for android-x86 build step
- S3 returned HTTP 500 when (mock) when running repack_10/10
Firefox 46.0b8
- http://hg.mozilla.org/releases/mozilla-beta/0334bcac4033
- Firefox build1 - had to rerun update verify jobs because they were cancelled. Perhaps because of bbb change?
Fennec 46.0b8
- http://hg.mozilla.org/releases/mozilla-beta/0334bcac4033
- Fennec build1 - several repacks failed due to inability to download packages, nthomas reran them
- Fennec download pages were 404 bug 1262460. Investigating why they were not pushed to mirrors or is this just a step missing from releaseduty steps