release tracking bugs for post-mortem:

Firefox 41.0b10

fennec only

firefox_source failed for Firefox 38.3.0esr build1
- cause: Source factory change from earlier today: http://hg.mozilla.org/build/buildbotcustom/rev/8c226d1b7a0b#l1.60
- resolution: backing out above cset, re-tagging, and re-triggering builder: http://buildbot-master73.bb.releng.usw2.mozilla.com:8001/builders/release-mozilla-esr38-firefox_source/builds/3
note: build2 is this week and just got shipped

release-runner failed on first kick-off
- cause: firefox-38.0.5b3.complete.mar didn't exist so partial versions were wrong. we needed: 40.0.3build1,39.0build6,40.0.2build1,41.0b9build1
- resolution: re-submitted with correct partials based on ADI
all fennec builds failed
- reason: bundleclone not working on linux/mock (https://bugzil.la/1191859) because:
  - a puppet change happened that day where tracebacks are logged if bundleclone fails (http://hg.mozilla.org/build/puppet/rev/04eb3e1f28c6)
  - a multi_locale automation step (make package) that will fail if it parses error lines (like tracebacks from part 2)
- resolution: http://hg.mozilla.org/build/buildbot-configs/rev/65ef8a7712e3
api-11 fennec repack failures (chunks 1,2,3,7,8,10)
- reason: because of a mock package failed to install:
  - 18:40:39 INFO - https://s3-us-west-2.amazonaws.com/mozilla-releng-mock-centos-us-west-2/mirrors/centos/6/latest/os/x86_64/GConf2-devel-2.28.0-6.el6.x86_64.rpm: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 500 Internal Server Error"
- resolution: the other chunks installed fine so it must have been a network issue between aws <-> mozilla. rebuilt the failed chunks
- follow-up: investigate network flows at the time of failure. might not be worth it.

needed to update the release sanity whitelist to include some exceptions
- do we need to run this for Firefox?
relman needed to leave out 41.0build3 from partials list (until we ship win64 on beta *and* release)
- would pushing to mirrors earlier help with this?
bug 1206983 - 42.0b1 l10n repacks failed to generate partial MARs
- happened for 41.0b1 as well
- Landed the patch on mozilla-beta's default, transplanted to the relbranch (there were new pushes on default), moved the tags, and pushed
- Rebuilt linux, linux64 and macosx64 l10n jobs (win32/win64 en-US were still in progress). "Rebuild" worked fine, no need to explicitly set script_repo_revision anymore
- pending patch for merge scripts

should changes like puppet deployments be allowed on mon/tues of release build week