Thunderbird 2.0.0.12:BuildNotes
Using Bootstrap (RELEASE_AUTOMATION_M7_1) but starting steps manually.
Contents
Build Engineers
nthomas
Version/config bump bug
Bonsai queries
Tags
Using the GECKO181_20080128_RELBRANCH.
Module | Tag | Pull date |
cvsroot/mozilla | THUNDERBIRD_2_0_0_12_RELEASE | GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST |
THUNDERBIRD_2_0_0_9_RC1 | GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST | |
l10n/l10n | THUNDERBIRD_2_0_0_12_RELEASE | GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST |
THUNDERBIRD_2_0_0_12_RC1 | GECKO181_20080128_RELBRANCH @ 2008-02-12 13:47 PST |
NB: Talkback not listed, we're using pre-build binaries (bug 403701). It was still tagged because this change is very new, and the automation needs remedial lessons.
Build data
See page history for BuildIDs of prior RCs
Type | Build ID | SHA1 | Push date | Build machine |
[Windows installer] | RC1:2008021305 | patrocles (cerberus-vm for l10n) | ||
[Mac compressed] | RC1:2008021304 | xserve02 | ||
[Linux compressed] | RC1:2008021305 | crazyhorse (karma for l10n) |
Detailed Notes
Automation
Bootstrap tag: RELEASE_AUTOMATION_M7_1
Setup before starting:
- used existing version bump bug to attached diff for tb-moz18-bootstrap.cfg
- ensure that machines have enough resources (bug 393274)
- On production-1.8-master, remove private and public areas:
- rm -rf /data/cltbld/firefox-*
- rm -rf /builds/tags/* - did not do this, plenty of space
- in /home/ftp/pub/firefox/nightly/, remove all except the 2.0.0.x-candidates directory for the previous release
- pull in 2.0.0.9 files with rsync -av stage.mozilla.org:/home/ftp/pub/thunderbird/nightly/2.0.0.9-candidates /data/ftp/pub/thunderbird/nightly/
- On the build machines (see table aboue), checked that there was sufficient working space (~ 1GB), and stopped tinderbox. Also verified there is space available on bm-xserve04 for l10n-verify.
- On prometheus-vm, checked that there was space for another update run (4GB)
- On production-1.8-master, remove private and public areas:
- on each machine, setup release automation using the tag above
cd /builds/release # or /cygdrive/X/builds/release cvs -nq up -r RELEASE_AUTOMATION_M7_1 # check for merges (M) or conflicts (C); if any found, check for local changes cvs -q up -r RELEASE_AUTOMATION_M7_1 cd configs cvs up -A # in case we need to make changes later cd .. rm bootstrap.cfg logs ln -s configs/tb-moz18-bootstrap.cfg bootstrap.cfg mkdir -p old-logs/logs.tb20012/rc1 ln -s old-logs/logs.tb20012/rc1 logs perl -c release
Some machines don't have the perl module Config::General so comment out
use Config::General;
at the top of Bootstrap/Step/PatcherConfig.pm. Only need this on prometheus-vm anyway, which has it.
Tag
On production-1.8-master in /builds/release:
./release -o Tag 2>&1 | tee logs/release-Tag-RC1.log
Had to come back to fix the 45 files in l10n/uk/editor, which had been tagged with MOZILLA_1_8_BRANCH by a localiser, rather than properly branched. The were all at rev 1.1 and had seen no changes on trunk, so it was straightforward to delete the existing tag, branch to MOZILLA_1_8_BRANCH, update to that, branch to the relbranch, and tag ..._RELEASE and ..._RC1.
Source
On production-1.8-master in /builds/release:
./release -o Source 2>&1 | tee logs/release-Source-RC1.log
Build
On each tinderbox, cd to /builds/release (prefix with /cygdrive/e/ on Windows) and
./release -o TinderConfig 2>&1 | tee logs/release-TinderConfig.log ./release -o Build 2>&1 | tee logs/release-Build.log
Needed to comment out lines 43-49 of Build.pm on Windows t'box, since Cygwin complains "Permissied denied" trying to execute those lines. The disks are already mounted in binmode.
Repack
On the tinderboxes in the table above,
./release -o Repack -e 2>&1 | tee logs/release-Repack-Execute.log ./release -o Repack -p 2>&1 | tee logs/release-Repack-Push.log ./release -o Repack -a 2>&1 | tee logs/release-Repack-Announce.log
Had to rerun Linux & Mac for the uk tagging issue (see above), but was able to manually checkout the files on the win32 box before it got to that locale.
After win32 l10n, had to set cerberus-vm back to text mode
mount -t -sc /cygdrive
so that tinderbox would run. This box was very slow, taking 7 hours to complete the repack.
Signing
As per the doc.
l10n-Verify
Ran this on bm-xserve05,
./release -o Repack -v 2>&1 | tee logs/release-Repack-Verify.log
Update Generation
Generated on production-prometheus-vm,
./release -o PatcherConfig 2>&1 | tee logs/release-PatcherConfig.log ./release -o Updates -e 2>&1 | tee logs/release-Updates-Execute.log ./release -o Updates -p 2>&1 | tee logs/release-Updates-Push.log
Push dirs were 20080218-Thunderbird-2.0.0.12 (& -test & -beta)
NB: DisableCompleteJump will need to be manually added to the rc block of the patcher config if we do an rc2, then regenerate the snippets.
QA found a problem with the details url - bug 418450. So we fix the config and regenerate the snippets:
cd /builds/updates/thunderbird-2.0.0.12/config cvs up cd ../patcher/temp/thunderbird/2.0.0.9-2.0.0.12 mkdir wrong-details-url mv aus2* wrong-details-url/ cd ../../.. ./patcher2.pl --create-patchinfo --app=thunderbird \ --config=../config/moz18-branch-patcher2.cfg 2>&1 | tee thunderbird-create-patchinfo.log
Do some quick verification
cd temp/thunderbird/2.0.0.9-2.0.0.12 diff -r aus2.test/ wrong-details-url/aus2.test/ | less diff -r aus2.beta/ wrong-details-url/aus2.beta/ | less diff -r aus2/ wrong-details-url/aus2/ | less
Only the detailsURL differs.
Push to aus2-staging as 20080219-Thunderbird-2.0.0.12{-test,-beta,}, then manually push 20080219-Thunderbird-2.0.0.12-test live.
Update Verify
Run on the tinderboxes
./release -o Updates -v 2>&1 | tee logs/release-Updates-Verify.log
On crazyhorse, verify failed because wget is too old. Removed --no-check-certificate argument in updates/verify.sh, common/download_mars.sh, and common/download_builds.sh, then reran.
patrocles doesn't have wget, so used production-pacifica-vm instead - needed to modify the win32_logDir from /cygdrive/e/... to /cygdrive/c/... in the bootstrap config.
Get expected FAIL's for 2.0b2/gu-IN/all platforms and 2.0b2/he/mac - no longer shipped. Also non-binary warnings for installed-chrome.txt and removed-files, which we no longer ignore.
Updates to Beta Channel
After QA gives "ok" to push to beta channel, on aus2-staging do:
# put snippets on beta $ screen $ sudo su - cltbld $ cd /opt/aus2/snippets/staging $ ~/bin/pushsnip 20080219-Thunderbird-2.0.0.12-beta
Stage
On production-1.8-master
./release -o Stage 2>&1 | tee logs/release-Stage.log
Backed up /data/cltbld/bin/verify-locales.pl before running this, and modified the three regex's on the file name (firefox -> thunderbird, Firefox -> Thunderbird). This is bug 409232.
Sign Installers
- NOTE - production-1.8-master is not used from here on
- pull stage:/data/cltbld/thunderbird-2.0.0.12/batch1/stage-signed/ to signing machine
- follow installer-signing-instructions here
- push signed bits back to same location on stage
- complete stage-merged:
# on stage cd /data/cltbld/thunderbird-2.0.0.12/ rsync -av batch1/mar/ stage-merged/ rsync -av batch1/stage-signed/ stage-merged/
- Create MD5 and SHA1 checksum files
# on stage cd /data/cltbld/thunderbird-2.0.0.12/stage-merged/ ~/bin/checksum-files .
- Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:thunderbird . chmod 644 *SUMS
Release
- get the last formal "go" from QA/Dev/website/IT/release-drivers
- push the stage-merged directory to the releases area (had to create target dir as root first):
# on stage rsync -av /data/cltbld/thunderbird-2.0.0.12/stage-merged/ /home/ftp/pub/thunderbird/releases/2.0.0.12/
- After pushing live, move forward the "latest" and "latest-2.0" symlinks.
cd /home/ftp/pub/thunderbird/releases/ rm latest ln -s 2.0.0.12 latest rm latest-2.0 ln -s 2.0.0.12 latest-2.0
- Update stage:/etc/rsyncd-mozilla-current.exclude - not critical that this happens immediately.
Update Bouncer
- Add entries to download.mozilla.org
Final Verification
Verify that releasetest points to valid bouncer links:
# this can be run from anywhere cvs co mozilla/testing/release cd mozilla/testing/release/updates cat moz18-thunderbird-*.cfg | sed 's/betatest/releasetest/' > update.cfg ./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging cd /opt/aus2/snippets/staging/20080219-Thunderbird-2.0.0.12 find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080219-Thunderbird-2.0.0.12-test/$a");'
Enable update channel
- Wait for QA to finish testing update snippets on releasetest channel, then enable update snippets on release channel
# on aus2-staging # put snippets on release/live channel $ screen $ sudo su - cltbld $ cd /opt/aus2/snippets/staging $ ~/bin/pushsnip 20080219-Thunderbird-2.0.0.12