Releases:Release Post Mortem:2016-01-06

From MozillaWiki
Jump to: navigation, search

Meeting Details

« previous week | index | next week »
< most recent | upcoming >


Release Duty

  • FF 44 cycle: jlund/rail (mtabara shadowing)
  • FF 45 cycle: mtabara

Misc

  • Firefox_44.0b3 and Firefox_44.0b5 skipped for holiday reasons
  • there will be NO "What's New Page" for FF 44 release

Firefox/Fennec_44.0b6 (jlund/rail)

  • post-release step still undone for Fennec 44.0b6. still pending QE signoff?
  • No failures of any type, some intermittent stuff:
    • failed at repack_9/10 on win64, automatic retry
    • all of the update verify linux failures were gtk related, hence ignored

Thunderbird 43.0b1 (jlund/rail/Callek/mtabara)

  • uplift to the release branch of part 1
  • release notes here
  • rail put a lot of work to fix the failing macosx repacks here. thank you!
  • Some intermittent failures came up:
    • build1 - one repack failure on linux from truncated complete mar, standalone_repack and repack_complete to recover
    • build1 - bug 1231174 all mac repacks failed
    • build2 - bug 1233577 Thunderbird builds failing with upload error (mach not found)
    • build3 - repack_7/10 and repack_6/10 failed on win32. TB QA seriously considering releasing in spite of the partial repacks
    • build3 - update_verify_beta_1/6 failed for Thunderbird 43.0b1 build3 on linux because all linux/linux64 update verification errors are GTK3 related

Ongoing

(not-yet)Firefox 43.0.4 (jlund/rail/mtabara)

  • current status: awaiting request to push it to mirrors
  • "Because of a high volume crash on Windows on 43.0.3 we disabled updates to 43.0.3 and are about to build, test, and I hope release 43.0.4"
  • Several failures along the way
    • build1: failed at repack_9/10 on win32 - timed out during cloning stuff, rerunning
    • build1: abandoned, starting build2 with "Backed out gdata patch, added sha-1 deprecation pref flip"
    • build2: abandoned as well. release automation pushed artifacts to build1 url location instead of build2
    • build2: there was an automation race condition where our masters tried pulling from hg.mozilla.org/build/buildbot-config and did not get served a recent push that included an incremental buildnum bump from 1 -> 2. This was likely due to the hg.m.o vcsreplicator issues that happened today and overlapped with the release and reconfig.
    • build2: we had an infrastructure failure with build2. Added a small ridealong patch to support download directories on multi-user machines
    • build3: started, in progress
    • build3: failed at firefox_antivirus, re-triggered; also failed in 4 out of 6 update_verify_release steps on win64, all retriggered

(not-yet)Thunderbird 38.5.1 (jlund/rail)

  • watershed release, has part2 patch
  • release notes here
  • current status: release-localtest channel looks good, awaiting request for release-cdntest. It's being hold off until the various upgrade paths using the waterfall are fully tested.
  • Some intermittent failures came up:
    • build1 - bug 1235071 ssh: symbolpush.mozilla.org: no address associated with name, invalid bug - wrong cset
    • build2 - bug 1226276 l10n taggging fails when running 'hg branches' against clean checkout of certain locale repos, rerunning

Action items

  • mtabara: we should run the post-release step on Fennec after the release gets pushed to Play Store. We don't need necessarily to wait for the sign-off
  • jlund: potential bump to 60 seconds of web heads for a better sync
  • jlund: follow-up on TB 38.5.1 to check the status with Wayne
  • mtabara: use bug numbers should you think info pasted in the wiki page is security-sensitive

Most Wanted bugs

Roundtable