Releases/Firefox 3.0.16/BuildNotes
From MozillaWiki
< Releases | Firefox 3.0.16
Contents
- 1 Build Engineer(s)
- 2 Bonsai queries
- 3 Tags
- 4 Notes
- 4.1 Build 1
- 4.2 Build 2
- 4.3 Build 3
- 4.3.1 Tag
- 4.3.2 Source
- 4.3.3 Build & Repack
- 4.3.4 Sign
- 4.3.5 L10nVerify
- 4.3.6 Generate Updates
- 4.3.7 Publish Updates to Test Channels (betatest & releasetest)
- 4.3.8 Update Verify
- 4.3.9 Stage
- 4.3.10 XULRunner
- 4.3.11 Push updates to beta channel
- 4.3.12 3.0.16 -> 3.5.6 Major Update Refresh
- 4.3.13 Sign Installers
- 4.3.14 Update Bouncer
- 4.3.15 Push to mirrors
- 4.3.16 Final Verification
- 4.3.17 Publish Updates to Release Channel
- 4.3.18 Stage
- 4.3.19 Sign Installers
- 4.3.20 Update Bouncer
- 4.3.21 Push to mirrors
- 4.3.22 Final Verification
- 4.3.23 Publish Updates to Release Channel
- 4.3.24 Release
- 4.3.25 Release matching XULRunner build
Build Engineer(s)
coop
Bonsai queries
- Build 1
- Build 2
- Build 3
Tags
Build 1:
| Module | Branch | Tag | Pull date |
| cvsroot/mozilla | HEAD | GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD1 FIREFOX_3_0_16_RELEASE | 2009-11-19 10:28 PDT (wrong timezone!) |
| l10n/l10n | HEAD | GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD1 FIREFOX_3_0_16_RELEASE | 2009-09-22 05:34 PDT |
Build 2:
| Module | Branch | Tag | Pull date |
| cvsroot/mozilla | HEAD | GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD2 FIREFOX_3_0_16_RELEASE | 2009-11-30 18:29 PST |
| l10n/l10n | HEAD | GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD2 FIREFOX_3_0_16_RELEASE | 2009-09-22 05:34 PDT |
Build 3:
| Module | Branch | Tag | Pull date |
| cvsroot/mozilla | HEAD | GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD3 FIREFOX_3_0_16_RELEASE | 2009-12-01 15:18 PST |
| l10n/l10n | HEAD | GECKO190_20091130_RELBRANCH FIREFOX_3_0_16_BUILD3 FIREFOX_3_0_16_RELEASE | 2009-09-22 05:34 PDT |
Notes
Build 1
- on fx-linux-1.9-slave2 deleted:
- /builds/verify/firefox-3.0.15
- /builds/verify/firefox-3015-355-major
- /builds/source/firefox-3.0.15
- /builds/data/cltbld/firefox-3.0.15
- /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend
- /builds/tinderbox/Fx-Mozilla1.9-l10n-Release//Linux_2.6.18-53.1.13.el5_Depend
- On fx-mac-1.9-slave2 deleted:
- /builds/verify/firefox-3.0.15
- /builds/verify/firefox-3015-354-major
- /builds/verify/firefox-3015-355-major
- On fx-win32-1.9-slave2 deleted:
- /e/xr19rel/WINNT_5.2_Depend
- /e/fx19l10nrel/WINNT_5.2_Depend
- /e/fx19rel/WINNT_5.2_Depend
- Available space on slaves:
- fx-linux-1.9-slave2 35 GB on /builds
- fx-mac-1.9-slave2 17 GB on /
- fx-win32-1.9-slave2 15.0 GB on D: / 14.1 GB on E:
- Kicked off automation:
buildbot sendchange --username=coop --master=localhost:9989 -m"Firefox 3.0.16build1 release" DOOOOOEEEEET!
Tag
Two aborted tag attempts prior to a successful one. Forgot to reconfig the master the first time, and then forgot to update the RELEASE_AUTOMATION_M15 tag on the bootstrap configs to point to the latest version.
Source
No problems.
Build & Repack
- Linux - No problems
- Mac - No problems
- Win32 - *** RESPIN CALLED FOR bug 529401 ***
Build 2
- on fx-linux-1.9-slave2 deleted:
- nothing
- On fx-mac-1.9-slave2 deleted:
- /builds/partners/partner-repacks/scripts/original_builds
- /builds/partners/partner-repacks/scripts/repacked_builds
- On fx-win32-1.9-slave2 deleted:
- /e/fx19l10nrel/WINNT_5.2_Depend
- /e/fx19rel/WINNT_5.2_Depend
- Available space on slaves:
- fx-linux-1.9-slave2 31 GB on /builds
- fx-mac-1.9-slave2 17 GB on /
- fx-win32-1.9-slave2 15.0 GB on D: / 14.1 GB on E:
- Update the bootstrap configs
- Check them in and tag them or tag will fail
- Kicked off automation:
buildbot sendchange --username=coop --master=localhost:9989 -m"Firefox 3.0.16build2 release" DOOOOOEEEEET!
Tag
- Forgot to tag the bumped configs, so the first tag run failed.
- Discovered that the build 1 timestamp specified PDT instead of PST, so we missed the last change when creating the relbranch
- interrupted source and en-US builds
- merged the change over to the branch, creating rev 3.216.4.1, and moved the FIREFOX_3_0_16_RELEASE and FIREFOX_3_0_16_BUILD2 tags onto that
- patched production-1.9-master:/build/buildbot/master.cfg and reconfiged
Index: master.cfg
===================================================================
RCS file: /cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v
retrieving revision 1.56
diff -u -r1.56 master.cfg
--- master.cfg 5 Aug 2009 03:43:58 -0000 1.56
+++ master.cfg 1 Dec 2009 03:11:29 -0000
@@ -449,13 +449,17 @@
haltOnFailure=1,
)
+from buildbot.process.factory import BuildFactory
+from buildbot.steps.dummy import Dummy
+dummyFactory = BuildFactory()
+dummyFactory.addStep(Dummy())
c['builders'].append(
{
'name': 'tag',
'slavename': 'fx-linux-1.9-slave2',
'category': 'release',
'builddir': 'tag',
- 'factory': tagFactory,
+ 'factory': dummyFactory,
},
)
- cleaned up fx-linux-1.9-slave2:/builds/source/firefox-3.0.16/batch-source/build2
- cleaned up /builds/config/firefox-3.0.16-build2 on all three slaves
- removed FIREFOX_3_0_16_BUILD2_l10n and FIREFOX_3_0_16_BUILD2 tags from mozilla/tools/tinderbox-configs/firefox
- stopped buildbot on win32 slave, clean up link/make/perl processes, start buildbot
- give another sendchange
buildbot sendchange --username=nthomas --master=localhost:9989 -m"Firefox 3.0.16build2 release - restart" DOOOOOEEEEET!
In retrospect, merging the change to the relbrnach and doing build 3 would have been much easier!
Source
No problems
Build & Repack
- Linux - No problems
- Mac - No problems
- Win32 - No problems
Sign
- Signing doc
- No problems
L10nVerify
- No problems
Generate Updates
- No problems
Publish Updates to Test Channels (betatest & releasetest)
Update Verify
- linux - No problems
- win32 - .chk files problems (EXPECTED)
- mac - No problems
Stage
No problems
*** RESPIN CALLED FOR bug 504862 ***
Build 3
- on fx-linux-1.9-slave2 deleted:
- /builds/source/firefox-3.0.16
- /builds/updates/firefox-3.0.16
- /builds/verify/firefox-3.0.16
- /builds/tags/FIREFOX_3_0_16_BUILD1
- /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend
- /builds/tinderbox/Fx-Mozilla1.9-l10n-Release/Linux_2.6.18-53.1.13.el5_Depend
- On fx-mac-1.9-slave2 deleted:
- nothing
- On fx-win32-1.9-slave2 deleted:
- /e/fx19l10nrel/WINNT_5.2_Depend
- /e/fx19rel/WINNT_5.2_Depend
- /e/xr19rel/WINNT_5.2_Depend
- Available space on slaves:
- fx-linux-1.9-slave2 29 GB on /builds
- fx-mac-1.9-slave2 24 GB on /
- fx-win32-1.9-slave2 15.0 GB on D: / 14.1 GB on E:
- reverted dummy factory change in master.cfg on production-1.9-master
- Updated the bootstrap configs
- Checked them in and tagged them with RELEASE_AUTOMATION_M15, otherwise tagging will fail.
- Kicked off automation:
buildbot sendchange --username=coop --master=localhost:9989 -m"Firefox 3.0.16build3 release" DOOOOOEEEEET!
- forgot to reconfig master, shutdown builds as quickly as possible, reconfig-ed , and then sent another sendchange
Tag
- No problems
Source
- No problems
Build & Repack
- 1st attempt - The aborted first run got as far as Tinderconfig before I could kill it. :/
- Linux - failed in Tinderconfig
- Mac - failed in Tinderconfig
- Win32 - slave lost???
- 2nd attempt, with Tinderconfig step commented out:
- Linux - No problems
- Mac - No problems
- Win32 - slow, but no problems
Sign
- Signing doc
- No problems
L10nVerify
- No problems
Generate Updates
- No problems
Publish Updates to Test Channels (betatest & releasetest)
- No problems
Update Verify
- linux - No problems
- win32 - .chk files problems (EXPECTED)
- mac - No problems
Stage
- solaris builds had been uploaded to contrib/ already, so rsync "failed" to sync contrib/ due to permissions issues, but everything else synced fine.
XULRunner
- kicked off by hand due to stage step "failing." No problems
Push updates to beta channel
# cltbld@aus2-staging.m.o # make sure scripts are up to date cd bin cvs update cd /opt/aus2/snippets/staging/ ~/bin/backupsnip 20091202-Firefox-3.0.16-beta
Then on official "go" email do pushsnip:
~/bin/pushsnip 20091202-Firefox-3.0.16-beta
3.0.16 -> 3.5.6 Major Update Refresh
- version bumps for patcher config and release/update
Snippet Generation
On fx-linux-1.9-slave2:
mkdir -p /builds/3.0.16-3.5.6-major-update/snippets cd /builds/3.0.16-3.5.6-major-update/snippets cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher cd patcher cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot # build tools ./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log # download mars ./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log # FIXME - patcher needs to see that the MARs that it thinks are partials # are there or else it will not attempt to generate patchinfo cd temp/firefox ln -s 3.5.6 3.0.16-3.5.6 cd ../.. # Create partial patches and snippets ./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log
Quick Verify
Check that releasetest = beta = release. New in 3.0.15 we now use a new dir for beta
cd temp/firefox/3.0.16-3.5.6
# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'
# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'
Push to AUS
New in 3.0.15 - 3 dirs to sync now instead of 2
cd /builds/3.0.16-3.5.6-major-update/snippets/patcher/temp/firefox/3.0.16-3.5.6 rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091202-Firefox-3.0.16-3.5.6-MU-test rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091202-Firefox-3.0.16-3.5.6-MU-beta rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20091202-Firefox-3.0.16-3.5.6-MU
Enable test snippets
ssh -i ~/.ssh/aus2 cltbld@aus2-staging cd /opt/aus2/snippets/staging
~/bin/backupsnip 20091202-Firefox-3.0.16-3.5.6-MU-test ~/bin/pushsnip 20091202-Firefox-3.0.16-3.5.6-MU-test
Update verify
Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:
mkdir -p /builds/verify/firefox-3016-356-major/ cd /builds/verify/firefox-3016-356-major/ hg clone http://hg.mozilla.org/build/tools # get patch to trawl "Only in" directories cd tools/release/common # on mac & linux curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=367544 | patch -p3 # on win32 wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=367544§ patch -p3 < patch # everyone ... cd ../updates platform=linux # or mac or win32 ./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log
Check logs for results - whatever is after "succeeded", known issues are here
Verify succeeded if in the results everything is the same as 3.0.16 -> 3.5.6 MU. aka, everything is OK
Sign Installers
- cleared some space on keymaster:
$ rm -rf xulrunner-1.9.0.13 xulrunner-1.9.0.14 xulrunner-1.9.1.[1-3] byob-3.5.3
- firefox-3.5.3* firefox-3.6a*
- Done manually on the day before release using these installer-signing-instructions here
On Stage complete stage-merged:
# on stage cd /data/cltbld/firefox-3.0.16/ rsync -av batch1/mar/ stage-merged/ rsync -av batch1/stage-signed/ stage-merged/
Create MD5 and SHA1 checksum files
# on stage cd /data/cltbld/firefox-3.0.16/stage-merged/ ~/bin/checksum-files .
Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox . chmod 644 *SUMS
Update Bouncer
Done.
Wait for official "push to mirrors" email...
Push to mirrors
Make sure that you use ffxbld for 'push to mirrors' , rather than cltbld (this is a change, but a desired one).
- push the stage-merged directory to the releases area. This can take a while, and can take even longer if there are contrib builds (solaris) in the subdirs already. For this reason, I sync the win32 builds first, and then perform a general rsync to catch everything after. This allows mirrors to start picking up the win32 builds (our most used platform) a little quicker:
# on stage mkdir -p /home/ftp/pub/firefox/releases/3.0.16/win32 rsync -av /data/cltbld/firefox-3.0.16/stage-merged/win32/ /home/ftp/pub/firefox/releases/3.0.16/win32/ rsync -av /data/cltbld/firefox-3.0.16/stage-merged/ /home/ftp/pub/firefox/releases/3.0.16/
- edit the exclude file (as cltbld) /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.16) and remove the previous release (3.0.15).
Final Verification
hg clone http://hg.mozilla.org/build/tools cd tools/release/updates cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg ./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
- Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
- No Problems
- Before pushing final updates, verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20091201-Firefox-3.0.16
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20091201-Firefox-3.0.16-test/$a");'
- this should have been 20091201-Firefox-3.0.16
Publish Updates to Release Channel
Start doing backupsnip a few hours at least before the release time
# cltbld@aus2-staging cd /opt/aus2/snippets/staging time ~/bin/backupsnip 20091201-Firefox-3.0.16
- this should have been 20091202-Firefox-3.0.16
- No problems
*** PROBLEM FOUND WITH UPDATES ON RELEASETEST ***
The solaris rsync problems in the Stage step were a bigger problem than I thought. The step exited early, and consequently build2 packages and mars were sent to mirrors instead of build3.
We deleted the releases/3.0.16 directory on stage.mozilla.org when we discovered this to prevent any further syncing. We realize that some users who checked manually may have found these installers. We plan to offer an update path from build2 sometime later this week.
In the meantime, we are going to start over from the Stage step (excluding the solaris builds), generate new snippets, re-sign installers, etc. We'll create a new product in bouncer, e.g. 3.0.16-real, and then do all the switching heroics in a few days once things calm down on the mirrors.
Stage
- landed exclusion for contrib dir
- tagged Bootstrap with RELEASE_AUTOMATION_M15_1
- landed master.cfg update to use new tag
- reconfig-ed production-1.9-master
- triggered Stage step from the waterfall.
- failed out when /builds was full; restarted after removing
/data/cltbld/firefox-3.0.16 /builds/tinderbox/Xr-Mozilla1.9-Release/Linux_2.6.18-53.1.13.el5_Depend/ /builds/3.0.15-3.5.5-major-update/patcher/temp/firefox/3.0.15/ftp/* /builds/3.0.16-3.5.6-major-update/snippets/patcher/temp/firefox/3.0.16/ftp/*
- No problems
Sign Installers
- moved stage-{unsigned,signed} to stage-{unsigned,signed}-bad_rsync_for_analysis
On Stage complete stage-merged:
# on stage cd /data/cltbld/firefox-3.0.16/ rsync -av batch1/mar/ stage-merged/ rsync -av batch1/stage-signed/ stage-merged/
Create MD5 and SHA1 checksum files
# on stage cd /data/cltbld/firefox-3.0.16/stage-merged/ ~/bin/checksum-files .
Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox . chmod 644 *SUMS
And merge in the solaris builds
rsync -av /pub/mozilla.org/firefox/nightly/3.0.16-candidates/build3/contrib/solaris_* contrib/
Update Bouncer
Changed the file locations from .../3.0.16/... to .../3.0.16-real/...
Push to mirrors
Make sure that you use ffxbld for 'push to mirrors', rather than cltbld (this is a change, but a desired one).
- push the stage-merged directory to the releases area. This can take a while, and can take even longer if there are contrib builds (solaris) in the subdirs already. For this reason, I sync the win32 builds first, and then perform a general rsync to catch everything after. This allows mirrors to start picking up the win32 builds (our most used platform) a little quicker:
# on stage mkdir -p /home/ftp/pub/firefox/releases/3.0.16-real/win32 rsync -av /data/cltbld/firefox-3.0.16/stage-merged/win32/ /home/ftp/pub/firefox/releases/3.0.16-real/win32/ rsync -av /data/cltbld/firefox-3.0.16/stage-merged/ /home/ftp/pub/firefox/releases/3.0.16-real/
- edit the exclude file (as cltbld) /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release (3.0.16-real) and remove the aborted release (3.0.16).
Final Verification
hg clone http://hg.mozilla.org/build/tools cd tools/release/updates cat moz19-firefox-{win32,mac,linux}.cfg | grep -v major | sed 's/betatest/releasetest/' > update.cfg ./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
- Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
- No Problems
- Before pushing final updates, verify that "release" and "releasetest" channel match:
# on aus2-staging
cd /opt/aus2/snippets/staging/20091202-Firefox-3.0.16
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20091202-Firefox-3.0.16-test/$a");'
Publish Updates to Release Channel
Backup snip was run above. Enable release snippets for minor update
time ~/bin/pushsnip 20091202-Firefox-3.0.16 real 2m52.488s user 0m0.178s sys 0m8.499s
Then again, waiting for go on MU
time ~/bin/pushsnip 20091202-Firefox-3.0.16-3.5.6-MU real 0m10.123s user 0m0.017s sys 0m0.570s
time ~/bin/pushsnip 20091202-Firefox-3.0.16-3.5.6-MU-beta real 0m16.796s user 0m0.024s sys 0m0.580s
Release
# ffxbld@stage cd /home/ftp/pub/firefox/releases rm latest-3.0 && ln -s 3.0.16-real latest-3.0
Release matching XULRunner build
XULRunner builds are produced by the automation, this is to groom them and push to the mirrors.
# cltbld@stage
VERSION=1.9.0.16
FX_VERSION=3.0.16
BUILD=3
mkdir -p ~/xulrunner-${VERSION}/batch1/stage-unsigned/{runtimes,source,sdk}
cd ~/xulrunner-${VERSION}/batch1/stage-unsigned
rsync -av --exclude=*sdk* --exclude=*info* \
/home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ runtimes/
rsync -av --include=*sdk* --exclude=* \
/home/ftp/pub/xulrunner/nightly/${VERSION}-candidates/build${BUILD}/ sdk/
cp -pv ~/firefox-${FX_VERSION}/stage-merged/source/firefox-${FX_VERSION}-source.tar.bz2 \
source/xulrunner-${VERSION}-source.tar.bz2
cd ~/xulrunner-${VERSION}/batch1
rsync -av stage-unsigned/ stage-signed/
Then create detached signatures per usual process (with PRODUCT=xulrunner, VERSION=1.9.0.16, don't need BUILD or TAG), skip the win32 signing section, do pgp, verify and upload. Back on stage:
cd ~/xulrunner-${VERSION}/batch1/stage-signed
cp ~/firefox-${FX_VERSION}/stage-merged/KEY .
~/bin/checksum-files .
chown -R cltbld:xulrunner .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ~/xulrunner-${VERSION}/batch1
rsync -nav stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/
If all is fine then
rsync -av stage-signed/* /home/ftp/pub/xulrunner/releases/${VERSION}/
(Mirror updating can take a bit, 10-15 mins this time)
Ping mfinkle or Mossop to update the links in these two wiki pages for the version change
