Releases/Thunderbird 15.0b1/BuildNotes
From MozillaWiki
< Releases
Be sure to take notes on manual steps and any problems you encounter.
Checklist
- Ship the l10n milestone DESKTOP MOBILE RELEASE BETA
- Set reserved_slaves DESKTOP MOBILE RELEASE BETA ESR
- Check to make sure you have all of the different types of slaves that you need DESKTOP MOBILE RELEASE BETA ESR
- Start the automation DESKTOP MOBILE RELEASE BETA ESR
- Reset reserved_slaves DESKTOP MOBILE RELEASE BETA ESR
- Run pushsnip DESKTOP RELEASE BETA ESR
Notes
tracking bug: bug 772810
buildbot master: 30
Build 1
- l10n changesets
- reserved slaves:
echo 8 > /builds/buildbot/build1/master/reserved_slaves'"
- clobbered release-comm-beta
- created candidates dir:
ssh -l tbirdbld -i ~/.ssh/tbirdbld_dsa stage.mozilla.org version=15.0b1 product=thunderbird cd /pub/mozilla.org/$product/nightly/ mkdir ../candidates/$version-candidates ln -s ../candidates/$version-candidates $version-candidates [tbirdbld@upload1.dmz.scl3 nightly]$ ls -l $version-candidates lrwxrwxrwx 1 tbirdbld thunderbird 31 Jul 17 10:17 15.0b1-candidates -> ../candidates/15.0b1-candidates
- updated & tagged
- reconfig
[cltbld@buildbot-master30 build1]$ source bin/activate (build1)[cltbld@buildbot-master30 build1]$ make update && make reconfig (cd buildbotcustom && /usr/bin/hg pull && /usr/bin/hg update -C) pulling from http://hg.mozilla.org/build/buildbotcustom searching for changes adding changesets adding manifests adding file changes added 4 changesets with 4 changes to 2 files (+1 heads) (run 'hg heads' to see heads, 'hg merge' to merge) 2 files updated, 0 files merged, 0 files removed, 0 files unresolved (cd buildbot-configs && /usr/bin/hg pull && /usr/bin/hg update -C) pulling from http://hg.mozilla.org/build/buildbot-configs searching for changes adding changesets adding manifests adding file changes added 9 changesets with 9 changes to 8 files (+1 heads) (run 'hg heads' to see heads, 'hg merge' to merge) 7 files updated, 0 files merged, 0 files removed, 0 files unresolved (cd tools && /usr/bin/hg pull && /usr/bin/hg update -C) pulling from http://hg.mozilla.org/build/tools searching for changes adding changesets adding manifests adding file changes added 3 changesets with 3 changes to 2 files (run 'hg update' to get a working copy) 2 files updated, 0 files merged, 0 files removed, 0 files unresolved (python buildbot-configs/update-master-json.py tools/buildfarm/maintenance/production-masters.json master/master_config.json) writing new master_config.json file to master/master_config.json cd master && /builds/buildbot/build1/bin/buildbot reconfig $PWD sending SIGHUP to process 20790 2012-07-17 10:49:19-0700 [-] loading configuration from /builds/buildbot/build1/master/master.cfg Never saw reconfiguration finish. (build1)[cltbld@buildbot-master30 build1]$ cd master && /builds/buildbot/build1/bin/buildbot reconfig $PWD sending SIGHUP to process 20790 Never saw reconfiguration finish.
- release sanity:
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u armenzg \ -V 15.0b1 --build-number 1 --branch comm-beta \ --release-config release-thunderbird-comm-beta.py --product thunderbird \ --dryrun localhost:9001
- reset reserved:
[cltbld@buildbot-master30 master]$ pwd -P /builds/buildbot/build1/master [cltbld@buildbot-master30 master]$ echo 0 >reserved_slaves [cltbld@buildbot-master30 master]$ cat reserved_slaves 0
- push snippets:
$ ssh aus3-staging.mozilla.org Last login: Wed Jul 11 13:45:06 2012 from vpn1.dmz.releng.scl3.mozilla.com [hwine@dp-ausstage01 ~]$ sudo su - tbirdbld [tbirdbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging [tbirdbld@dp-ausstage01 staging]$ ~/bin/pushsnip Thunderbird-15.0b1-build1
- we will have to push again since we hit the trigger too early - WIP
- pushed again
$ ssh aus3-staging.mozilla.org Last login: Wed Jul 11 13:45:06 2012 from vpn1.dmz.releng.scl3.mozilla.com [hwine@dp-ausstage01 ~]$ sudo su - tbirdbld [tbirdbld@dp-ausstage01 ~]$ cd /opt/aus2/snippets/staging [tbirdbld@dp-ausstage01 staging]$ ~/bin/pushsnip Thunderbird-15.0b1-build1
- re-triggered update verification failed jobs
XXX: check those jobs in the morning
Problems Encountered
- release sanity errors: mozconfig and L10n
2012-07-17 10:50:35,651 : WARNING : if test "$IS_NIGHTLY" != ""; then not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']! 2012-07-17 10:50:35,651 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: if test "$IS_NIGHTLY" != ""; then 2012-07-17 10:50:35,651 : WARNING : ac_add_options --disable-auto-deps not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']! 2012-07-17 10:50:35,652 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: ac_add_options --disable-auto-deps 2012-07-17 10:50:35,652 : WARNING : fi not in win32 ['ac_add_options --enable-update-channel=nightly', 'ac_add_options --enable-signmar', 'ac_add_options --enable-js-diagnostics', '. $topsrcdir/configs/mozilla2/win32/include/choose-make-flags', 'mk_add_options MOZ_MAKE_FLAGS=-j1']! 2012-07-17 10:50:35,652 : ERROR : found in mail/config/mozconfigs/win32/nightly but not in mail/config/mozconfigs/win32/release: fi 2012-07-17 10:50:37,865 : ERROR : Error verifying mozconfigs 2012-07-17 10:51:04,515 : ERROR : l10n_changesets and shipped_locales differ on locales: ['el', 'zh-TW', 'ja-JP-mac', 'ta-LK', 'vi', 'ca', 'it', 'cs', 'pa-IN', 'ar', 'bn-BD', 'nb-NO', 'eu', 'et', 'gl', 'es-ES', 'id', 'en-GB', 'ru', 'sr', 'nl', 'sl', 'tr', 'lt', 'rm', 'ro', 'is', 'pl', 'be', 'fr', 'bg', 'sv-SE', 'ast', 'hr', 'de', 'da', 'pt-BR', 'br', 'fi', 'hu', 'ja', 'he', 'hy-AM', 'pt-PT', 'es-AR', 'sq', 'ko', 'nn-NO', 'ga-IE', 'sk', 'zh-CN', 'si', 'gd', 'uk', 'fy-NL']
- bhearsum told me I can ignore the mozconfig errors and file a follow up bug (used "-m" to skip it)
- shipped-locales had been migrated only with "en-US" in it
- I bumped the comm-beta revision and re-tagged
- One L10n repack failed
command: START command: hg update -C -r THUNDERBIRD_15_0b1_RELEASE command: cwd: mozilla-beta/ru command: output: abort: data/.hgtags.i@b44e93cb42b2: no match found! command: ERROR Traceback (most recent call last): File "/builds/slave/tb-rel-c-beta-lnx-rpk-5/scripts/scripts/l10n/../../lib/python/util/commands.py", line 42, in run_cmd return subprocess.check_call(cmd, **kwargs) File "/tools/python-2.5.1/lib/python2.5/subprocess.py", line 461, in check_call raise CalledProcessError(retcode, cmd) CalledProcessError: Command '['hg', 'update', '-C', '-r', 'THUNDERBIRD_15_0b1_RELEASE']' returned non-zero exit status 255 command: END (0.37s elapsed)
- re-trigger "ru" with standalone repack (script_repo_revision: THUNDERBIRD_15_0b1_RELEASE, release_config: mozilla/release-thunderbird-comm-beta.py and locale: ru)
- trigger "repack complete"
- I pushed the snippets by mistake early, to shut off updates did:
chmod 700 /opt/aus2/incoming/3/Thunderbird
- we had to use the full back up nightly of the snippets since the incremental misses old releases, this is the short version of how we fixed them up
cd /opt/aus2/snippets/backup mkdir 15.0b1-restore2 && cd mkdir 15.0b1-restore2 tar --exclude=Firefox -jvxf ../20120718-nightly-1.tar.bz2 rsync -av --delete --exclude=1\.5* --exclude=2.0* --exclude=3.0* Thunderbird/ /opt/aus2/incoming/3/Thunderbird/ > /home/tbirdbld/changes.txt cd /opt/aus2/snippets/staging/ ~/bin/pushsnip Thunderbird-15.0b1-build1-test
We originally tried to restore from the backupsnip that was run prior to the 15.0b1 test snippets were pushed, but that ended up deleting a bunch of snippets that weren't in it due to a bad rsync. Due to this, we ended up using the full backup to restore as described above. There was also a lot of "rsync -n" commands run to verify things prior to doing them for real, not mentioned here for the sake of brevity.