Releases/Thunderbird 16.0.1/BuildNotes
Contents
Checklist
-
Set reserved_slaves DESKTOP MOBILE RELEASE BETA ESR -
Check to make sure you have all of the different types of slaves that you need DESKTOP MOBILE RELEASE BETA ESR -
Start the automation DESKTOP MOBILE RELEASE BETA ESR -
Reset reserved_slaves and unlock slaves DESKTOP MOBILE RELEASE BETA ESR -
Push to mirrors DESKTOP RELEASE -
Run pushsnip DESKTOP RELEASE BETA ESR
Bugs hit
- bug 799806
- need extra mozconfig line --enable-stdcxx-compat to build libmar. bhearsum fixed
- only fixed Firefox the first time round, was a new kind of broken on Thunderbird. nthomas fixed
- bug 799347 - Investigate packet loss within SCL3
- network issues caused repacks to fail
- Note for post-mortem Ask relman to advise IT of chemspills.
- bug 800277 - Update generation has problems when latest-1 --> latest partial isn't generated
- bug 800338 - win32 'id' locale for thunderbird 16.0.1 has wrong checksums filesizes
Notes
- Tracking bug bug 800023
- Release engineer bhearsum, kmoir, nthomas
- Buildbot master: bm34
Build 1
- Updated de revision to that used for 16.0 respin
- Set reserved slaves to 2
- Moved 2 64-bit linux ix slaves, 1 64-bit windows slave, and 2 32-bit windows slaves to bm34 to make sure we have enough of the right slaves to do the release.
- Tagged buildbot-configs, buildbotcustom, and tools with THUNDERBIRD_16_0_1_{RELEASE,BUILD1}
- Updated and reconfiged bm34
- Ran release sanity in dry run mode:
cd /builds/buildbot/build1/master source ../bin/activate PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u nthomas \ -V 16.0.1 --build-number 1 --branch comm-release \ --release-config release-thunderbird-comm-release.py --product thunderbird \ --l10n-dashboard-version=16 --dryrun localhost:9001 2012-10-10 14:51:21,104 : INFO : Comparing l10n changesets on dashboard https://l10n.mozilla.org/shipping/l10n-changesets?ms=tb16 to on-disk l10n-changesets_thunderbird-release ... 2012-10-10 14:51:21,151 : ERROR : locale de revisions not matching: 435ffc2817d3 (config) vs. 5eb2fb1a7155 (dashboard) 2012-10-10 14:51:21,151 : ERROR : Error verifying l10n dashboard changesets
so the dashboard hasn't got the de fix.
- Ran without --dryrun, and with -l to skip l10n check, to start the release
Fix repacks
Due to bug 799806, we need to add --enable-stdcxx-compat to our Linux l10n mozconfigs. This was done in https://hg.mozilla.org/build/buildbot-configs/rev/0029772efc8d. That revision was retagged, and 'rebuild' was used on all of the failed repacks (linux + linux64) to get them going again.
Also needed to handle tier_base and tier_nspr properly for thunberbird, bug 799806 again.
Repacks failed due to network issues in bug 799347 (scl3 core router problem). After enough retries and IT work around they succeeded.
Updates
Update generation builder went red because we don't really support not generating a partial from the previous release, bug 800277.
Generate 16.0 -> 16.0.1 complete update
The locales list didn't change between 15.0.1 and 16.0 so we can just grab the 15.0.1 snippets, munge the buildID and zap the partial.txt files.
# tbirdbld@aus3-staging cd /opt/aus2/snippets/staging/ mkdir -p Thunderbird-16.0.1-build1-16.0-test/Thunderbird rsync -av ../pushed/Thunderbird-16.0.1-build1-test/Thunderbird/15.0.1/ \ Thunderbird-16.0.1-build1-16.0-test/Thunderbird/16.0/ for d in Thunderbird-16.0.1-build1-16.0-test/Thunderbird/16.0/*; do mv -v $d/20120907140327 $d/20121005141928 done find Thunderbird-16.0.1-build1-16.0-test/Thunderbird/16.0/ -type f -name partial.txt -exec rm {} \; pushsnip Thunderbird-16.0.1-build1-16.0-test
Release snippets, from the same working location
rsync -av Thunderbird-16.0.1-build1/Thunderbird/15.0.1/ \ Thunderbird-16.0.1-build1/Thunderbird/16.0/ for d in Thunderbird-16.0.1-build1/Thunderbird/16.0/*; do mv -v $d/20120907140327 $d/20121005141928 done find Thunderbird-16.0.1-build1/Thunderbird/16.0/ -type f -name partial.txt -exec rm {} \;
Generate update verify configs and trigger
On my own machines
mkdir tmp; cd tmp hg clone http://hg.mozilla.org/build/tools wget -O shipped-locales http://hg.mozilla.org/releases/comm-release/raw-file/THUNDERBIRD_16_0_1_RELEASE/mail/locales/shipped-locales
Apply this patch
diff --git a/lib/python/release/updates/patcher.py b/lib/python/release/updates/patcher.py --- a/lib/python/release/updates/patcher.py +++ b/lib/python/release/updates/patcher.py @@ -47,5 +47,5 @@ class PatcherConfig(dict): # assume that all of the fromVersions in the past-update lines are # versions that should have update paths to the latest on all channels. - return tuple([self['current-update']['from']] + [v[0] for v in self['past-update']]) + return tuple(set([self['current-update']['from']] + [v[0] for v in self['past-update']])) def getOptionalAttrs(self, version):to resolve the locale error. Then generate the configs with
python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozRelease-thunderbird-mac64.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASE python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform win32 --output tools/release/updates/mozRelease-thunderbird-win32.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASE python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform linux64 --output tools/release/updates/mozRelease-thunderbird-linux64.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASE python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozRelease-thunderbird-branch-patcher2.cfg --platform linux --output tools/release/updates/mozRelease-thunderbird-linux.cfg --release-config-file mozilla/release-thunderbird-comm-release.py -b http://hg.mozilla.org/build/buildbot-configs --channel betatest -t THUNDERBIRD_16_0_1_RELEASEThen add the 16.0 lines we want by doing this in each config file
- copy the two 15.0 lines up to the top of the file, and make some modifications ...
- change to release="16.0"
- change the build_id to 20121005141928
- change the 15.0's to 16.0's in the from="/thunderbird..." vars
- move aus_server and to definitions to the first line
Commit and move THUNDERBIRD_16_0_1_{RELEASE,BUILD1}_RUNTIME tags.
Then forced update verify builders (from local machine)
for p in linux linux64 macosx64 win32; do for i in 1 2 3 4; do curl "http://buildbot-master34.build.mozilla.org:8001/builders/release-comm-release-${p}_update_verify_${i}%2F4/force" > /dev/null sleep 5 done done
Issues with id locale on win32
eg
FAIL: complete from https://aus3.mozilla.org/update/1/Thunderbird/16.0/20121005141928/WINNT_x86-msvc/id/betatest/update.xml?force=1 wrong size FAIL: update.xml size: 22681695 FAIL: actual size: 22681738 complete from https://aus3.mozilla.org/update/1/Thunderbird/15.0.1/20120907140327/WINNT_x86-msvc/id/betatest/update.xml?force=1 wrong size update.xml size: 22681695 actual size: 22681738 download_mars returned non-zero exit code: 1 partial from https://aus3.mozilla.org/update/1/Thunderbird/15.0.1/20120907140327/WINNT_x86-msvc/id/betatest/update.xml?force=1 wrong size update.xml size: 7565330 actual size: 7565337 download_mars returned non-zero exit code: 1
bhearsum did some investigating and couldn't find any obvious cause. bug 800338 was filed to track it down. In the meantime, did the following to fix us up by hand:
# tbirdbld@aus3-staging cd /opt/aus2/snippets/staging rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/ Thunderbird-16.0.1-build1-test-fixed-id/ # values on the left side of the replacements are from the checksums file on the slave; values on the right are from the checksums on ftp find Thunderbird-16.0.1-build1-test-fixed-id -type f -regex '.*/id/.*' -exec sed -i -e 's/22681695/22681738/' -e 's/43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1/fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664/' -e 's/7565330/7565337/' -e 's/3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78/9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e/' {} \; rsync -av Thunderbird-16.0.1-build1/ Thunderbird-16.0.1-build1.bak/ find Thunderbird-16.0.1-build1 -type f -regex '.*/id/.*' -exec sed -i -e 's/22681695/22681738/' -e 's/43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1/fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664/' -e 's/7565330/7565337/' -e 's/3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78/9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e/' {} \;
Output from some quick verifications:
diff -Naur /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/Thunderbird/ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/ | grep '^++' +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0/WINNT_x86-msvc/20120129064235/id/betatest/complete.txt 2012-10-11 06:19:42.652817000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0/WINNT_x86-msvc/20120129064235/id/releasetest/complete.txt 2012-10-11 06:19:42.656822000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.1/WINNT_x86-msvc/20120208125618/id/betatest/complete.txt 2012-10-11 06:19:41.151731000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.1/WINNT_x86-msvc/20120208125618/id/releasetest/complete.txt 2012-10-11 06:19:41.155731000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.2/WINNT_x86-msvc/20120216022751/id/betatest/complete.txt 2012-10-11 06:19:41.766769000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/10.0.2/WINNT_x86-msvc/20120216022751/id/releasetest/complete.txt 2012-10-11 06:19:41.771766000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0/WINNT_x86-msvc/20120312212756/id/betatest/complete.txt 2012-10-11 06:19:44.587936000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0/WINNT_x86-msvc/20120312212756/id/releasetest/complete.txt 2012-10-11 06:19:44.591934000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0.1/WINNT_x86-msvc/20120327131646/id/betatest/complete.txt 2012-10-11 06:19:43.554883000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/11.0.1/WINNT_x86-msvc/20120327131646/id/releasetest/complete.txt 2012-10-11 06:19:43.559878000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0/WINNT_x86-msvc/20120420153905/id/betatest/complete.txt 2012-10-11 06:19:46.335039000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0/WINNT_x86-msvc/20120420153905/id/releasetest/complete.txt 2012-10-11 06:19:46.339037000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0.1/WINNT_x86-msvc/20120428123112/id/betatest/complete.txt 2012-10-11 06:19:45.443986000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/12.0.1/WINNT_x86-msvc/20120428123112/id/releasetest/complete.txt 2012-10-11 06:19:45.448984000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0/WINNT_x86-msvc/20120604214345/id/betatest/complete.txt 2012-10-11 06:19:48.457166000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0/WINNT_x86-msvc/20120604214345/id/releasetest/complete.txt 2012-10-11 06:19:48.462172000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0.1/WINNT_x86-msvc/20120614161456/id/betatest/complete.txt 2012-10-11 06:19:47.281099000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/13.0.1/WINNT_x86-msvc/20120614161456/id/releasetest/complete.txt 2012-10-11 06:19:47.286093000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/14.0/WINNT_x86-msvc/20120713141924/id/betatest/complete.txt 2012-10-11 06:19:49.827247000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/14.0/WINNT_x86-msvc/20120713141924/id/releasetest/complete.txt 2012-10-11 06:19:49.832249000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0/WINNT_x86-msvc/20120824172341/id/betatest/complete.txt 2012-10-11 06:19:52.509405000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0/WINNT_x86-msvc/20120824172341/id/releasetest/complete.txt 2012-10-11 06:19:52.514404000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/betatest/complete.txt 2012-10-11 06:19:51.195324000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/betatest/partial.txt 2012-10-11 06:19:51.198326000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/releasetest/complete.txt 2012-10-11 06:19:51.202332000 -0700 +++ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/releasetest/partial.txt 2012-10-11 06:19:51.205331000 -0700 diff -Naur /opt/aus2/snippets/staging/Thunderbird-16.0.1-build1.bak/Thunderbird/ Thunderbird-16.0.1-build1/Thunderbird/ | grep '^++' +++ Thunderbird-16.0.1-build1/Thunderbird/10.0/WINNT_x86-msvc/20120129064235/id/release/complete.txt 2012-10-11 06:29:26.041502000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/10.0.1/WINNT_x86-msvc/20120208125618/id/release/complete.txt 2012-10-11 06:29:24.568416000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/10.0.2/WINNT_x86-msvc/20120216022751/id/release/complete.txt 2012-10-11 06:29:25.353462000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/11.0/WINNT_x86-msvc/20120312212756/id/release/complete.txt 2012-10-11 06:29:27.495594000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/11.0.1/WINNT_x86-msvc/20120327131646/id/release/complete.txt 2012-10-11 06:29:26.740552000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/12.0/WINNT_x86-msvc/20120420153905/id/release/complete.txt 2012-10-11 06:29:28.945675000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/12.0.1/WINNT_x86-msvc/20120428123112/id/release/complete.txt 2012-10-11 06:29:28.188630000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/13.0/WINNT_x86-msvc/20120604214345/id/release/complete.txt 2012-10-11 06:29:30.463765000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/13.0.1/WINNT_x86-msvc/20120614161456/id/release/complete.txt 2012-10-11 06:29:29.664720000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/14.0/WINNT_x86-msvc/20120713141924/id/release/complete.txt 2012-10-11 06:29:31.308823000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/15.0/WINNT_x86-msvc/20120824172341/id/release/complete.txt 2012-10-11 06:29:32.615892000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/release/complete.txt 2012-10-11 06:29:32.012863000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/15.0.1/WINNT_x86-msvc/20120907140327/id/release/partial.txt 2012-10-11 06:29:32.016859000 -0700 +++ Thunderbird-16.0.1-build1/Thunderbird/16.0/WINNT_x86-msvc/20121005141928/id/release/complete.txt 2012-10-11 06:29:33.429941000 -0700
diff -Naur /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/Thunderbird/ Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/ | grep '^[-+]' | grep -v '+++' | grep -v '\-\-\-' -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78 -size=7565330 +hashValue=9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e +size=7565337 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78 -size=7565330 +hashValue=9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e +size=7565337 diff -Naur /opt/aus2/snippets/staging/Thunderbird-16.0.1-build1.bak/Thunderbird/ Thunderbird-16.0.1-build1/Thunderbird/ | grep '^[-+]' | grep -v '+++' | grep -v '\-\-\-' -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738 -hashValue=3e5cc347bce240dc76d357fc892b92c89d6012f60c497ab468cc64a95f45e300e459dc2a92817e431a2e5b4e114858d2641e86685170c936174d0ccaeac00b78 -size=7565330 +hashValue=9821e09c802d8fa9cb2fb0532bd46830aa315833df972c122ef71ec91c8c5c8fe1964fc06e0c01913a91fe2e70286572140e17bfb3fb9fdf5b7d033d7853d27e +size=7565337 -hashValue=43ad0ebfde1b19aa538595ef4f81b5d6b12ffd3a8b1c73ffb065c9d33173c4f3715afd6763ecdce2d1c65a4f12d190f4c04d4eca6f0304dca991a0de3253d1a1 -size=22681695 +hashValue=fc694b6bbcc0bd58a3644228ed826168b2a8edcd3125a9a21f7f2f7e70de75824b4acce8a886d39bde9f962b5cd6e714f6ebcc9567a2e6c2c435fd0332450664 +size=22681738
Pushed the updated test snippets:
~/bin/pushsnip Thunderbird-16.0.1-build1-test-fixed-id
And retriggered win32 update verify through the "rebuild" button.
The first update verify chunk failed again, but only for the "16.0" update. This is because I forgot to take into account the manually generated 16.0 updates. To fix:
# tbirdbld@aus3-staging cd /opt/aus2/snippets/staging/ mkdir -p Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird rsync -av --include=WINNT_x86-msvc --include=20120907140327 --include=id --include=betatest --include=releasetest --include=*.txt \ --exclude=* ../pushed/Thunderbird-16.0.1-build1-test-fixed-id/Thunderbird/15.0.1/ \ Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/ mv -v Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/WINNT_x86-msvc/20120907140327 Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/WINNT_x86-msvc/20121005141928 find Thunderbird-16.0.1-build1-16.0-test-fixed-id/Thunderbird/16.0/ -type f -name partial.txt -exec rm -v {} \; ~/bin/pushsnip Thunderbird-16.0.1-build1-16.0-test-fixed-id
Manually triggered update verify chunk #1 again.
No need to adjust the release snippets, because they already had 16.0 snippets in them. To be paranoid, used the snippet compare script to verify that releasetest and release matched:
# tbirdbld@aus3-staging cd ~/ # "repush" snippets into a non-live directory so we have something to compare against. rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test/ Thunderbird-16.0.1-build1-test-complete/ rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-16.0-test/ Thunderbird-16.0.1-build1-test-complete/ rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-test-fixed-id/ Thunderbird-16.0.1-build1-test-complete/ rsync -av /opt/aus2/snippets/pushed/Thunderbird-16.0.1-build1-16.0-test-fixed-id/ Thunderbird-16.0.1-build1-test-complete/ hg clone http://hg.mozilla.org/build/tools wget -O- https://hg.mozilla.org/build/tools/archive/tip.tar.gz | gunzip | tar -xv cd tools-eb240f672142/release bash compare-channel-snippets.sh /opt/aus2/snippets/staging/Thunderbird-16.0.1-build1 release ~/Thunderbird-16.0.1-build1-test-complete/ releasetest
...and then retriggered ALL update verify builders because I'm super paranoid:
for p in linux linux64 macosx64 win32; do for i in 1 2 3 4; do curl "http://buildbot-master34.build.mozilla.org:8001/builders/release-comm-release-${p}_update_verify_${i}%2F4/force" > /dev/null sleep 5 done done
Also had to update the checksums files. To do this, manually edited /pub/mozilla.org/thunderbird/nightly/16.0.1-candidates/build1/win32/id/thunderbird-16.0.1.checksums on upload1 to have correct values. With that fixed, used "force build" to start the "checksums" builder again to regenerate the big *SUMS files + signatures. The builder burned when trying to create contrib dirs again, but it had already uploaded the new *SUMS files + sigs, so that's okay. Ran the following to sync the new files to releases/
# tbirdbld@upload1 rsync -av --include=*SUMS* --exclude=* /pub/mozilla.org/thunderbird/nightly/16.0.1-candidates/build1/ /pub/mozilla.org/thunderbird/releases/16.0.1/
push to mirrors
initiated the following by forcing build on bm34 release-comm-release-push_to_mirrors start update monitoring release-comm-release-macosx64_final_verification release-comm-release-win32_final_verification release-comm-release-linux64_final_verification release-comm-release-linux_final_verification ready for release ready for releasetest
pushsnip
[tbirdbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging [tbirdbld@dp-ausstage01 staging]$ ~/bin/pushsnip Thunderbird-16.0.1-build1
Unthrottle
Standard8 asked us to unthrottle right after pushing. this patch was landed and retagged with AUS2_PRODUCTION. The push was done in bug 800473
Manually stop TriggerBouncerCheck
For some reason, probably related to Bouncer being broken and then fixed (bug 800457, this master kept firing the post-"ready for rel test" builders over and over. To fix, manhole'd in and ran the following:
master.scheduler_manager.namedServices["release-comm-release-ready-for-release"].stopLoop() master.scheduler_manager.namedServices["release-comm-release-ready-for-rel-test"].stopLoop('shut up already!')
This didn't work. Rail started poking at it afterwards.
A brute force attack via Manhole:
>>> print "\n".join(["%s %s" % (i, str(x)) for i, x in enumerate(master.allSchedulers()) if "TriggerBouncerCheck" in str(x)]) 13 <buildbotcustom.scheduler.TriggerBouncerCheck instance at 0x2aaad8dc0a70> 27 <buildbotcustom.scheduler.TriggerBouncerCheck instance at 0x2aaad4249b00> >>> master.allSchedulers()[13].loop = None >>> master.allSchedulers()[27].loop = None
I also set the master for a graceful shutdown to cleanup the brute force hacks.