Releases:Release Post Mortem:2015-09-03
From MozillaWiki
release tracking bugs for post-mortem:
- Bug 1198816 - tracking bug for build and release of firefox 40.0.3
- Bug 1199390 - tracking bug for build and release of Thunderbird 41.0
- Bug 1188493 - tracking bug for build and release of Firefox and Fennec 41.0
Contents
Firefox 40.0.3
- Win32 repack 13/20 failed with what looks like sh crashing during configure
- Rebuilt the chunk
- Updates step failed with:
release.updates.patcher.PatcherConfigError: Found multiple past-updates with duplicate to/from versions: ['40.0', '40.0.2', ['betatest', 'releasetest', 'release']]
- Fixed with https://bugzilla.mozilla.org/attachment.cgi?id=8652919&action=edit and a move of the FIREFOX_40_0_3_{RELEASE,BUILD1} tags to the new rev. A rebuild then succeeded.
Firefox 41.0b2
- failed at firefox_beta_updates on update_verify
- resolution: patch patcher configs: https://bugzilla.mozilla.org/show_bug.cgi?id=1188493#c19
- failed at repack_9/10 on win64
- resolution: was a slow slave and timed out in configure step. nick re-ran this builder
- [fennec only] bouncer submission and post-release steps failed
- resolution: ignore
- further action: solve https://bugzil.la/1193613
- update_verify_beta_3/6 failed for Firefox 41.0b2 build1 on win32 <- hung for 10 hours
- resolution: rebuild because it slave was in bad condition (couldn't delete previous file artifacts)
- further action: always sanity check that there is a "ready to ship to the beta channel" email after relman asks for 'push to beta channel'
Firefox 41.0b3
- win32 build failed because of slow win slave trying to remove old buildirs (retriggered)
- win{32,64}_xul failed because of https://bugzil.la/1170212 (win32_xul didn't even make it that far because it too timed out trying to remove old builddirs)
- win64 repack_5/10 failed due to the en-US firefox.zip being only ~9MB instead of 50MB. nthomas reran
- Firefox 41.0b3 build1: failed at update_verify_beta_4/6 on win32. jlund re-ran. https://bugzilla.mozilla.org/show_bug.cgi?id=1188493#c23
Firefox 41.0b4
- win{32,64} xulrunner failed to build/post-release
- win64 needed a buildspace bump of s/12/14
- bouncer single locale submission still broken. should be fixed in beta5 via https://bugzil.la/1193613
Firefox 41.0b5
- bouncer submit (again) bc single locale: tracking https://bugzilla.mozilla.org/show_bug.cgi?id=1193613#c27
Thunderbird 41.0b1
- all the repacks failed because they needed 'export' in make config target
- resolved with https://bugzilla.mozilla.org/show_bug.cgi?id=1193620#c5
- repacks are now failing because of a build sys error. something to do with lightning. tb folks are debugging
- looks like it's been resolved: https://bugzilla.mozilla.org/show_bug.cgi?id=1199390#c6
- expect a build2 soon
Roundtable
- [jlund] TCW - we accidentally kicked off a bunch of release jobs. anything we can do about blocking that in the future?
- https://dxr.mozilla.org/build_buildbotcustom/source/scheduler.py#446
- action: jlund to file bug
- action: file bug to chmod -R a-w candidates / release directories (excepting contrib)
- ftp -> archive.m.o (s3) seemed to not cause any issues
- anything to look out for in future?