Releases/Firefox 3.6.20/BuildNotes
Contents
- 1 Notes About Releasing
- 2 Bugs hit
- 3 Build Engineers
- 4 Signed-off Revision(s)
- 5 L10N changesets
- 6 Tags
- 7 Build data
- 8 Notes
- 8.1 Build 1
- 8.1.1 E-mail Metrics
- 8.1.2 Tag
- 8.1.3 Start autosign
- 8.1.4 Bouncer Submitter
- 8.1.5 Source
- 8.1.6 Build
- 8.1.7 Unittests / Talos
- 8.1.8 XULRunner
- 8.1.9 Repacks
- 8.1.10 Firefox Signing
- 8.1.11 L10N verify
- 8.1.12 Updates
- 8.1.13 3.6.18 -> 3.6.20 Partials
- 8.1.14 Major Update
- 8.1.15 Socorro Product Data Updates
- 8.1.16 Check permissions / AV scan
- 8.1.17 Reset reserved slaves
- 8.1.18 Push to beta
- 8.1.19 Push to internal mirrors
- 8.1.20 Push XULRunner to Mirrors
- 8.1.21 Update XULRunner wiki page
- 8.1.22 Push to external mirrors
- 8.1.23 Final verification
- 8.1.24 Push to Release Channel
- 8.1.25 Update Symlinks
- 8.1.26 Push 3.6.20 -> 5.0.1 MUs
- 8.1.27 3.6.20 -> 6.0 MU
- 8.1 Build 1
Notes About Releasing
Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed
Bugs hit
- bug 676534 - Clobberer broken for Major Update builders
-
bug 676013 - L10nVerify is giving a setupBuild exception
Build Engineers
bhearsum - bug 675730
Signed-off Revision(s)
Build 1: 48b6fb82a960
L10N changesets
Tags
Build # | Branch, Tags | Changeset |
1 | GECKO19220_2011080310_RELBRANCH | 48b6fb82a960 |
FIREFOX_3_6_20_BUILD1 FIREFOX_3_6_20_RELEASE | fec2e8213a9d |
Build data
Firefox
Build # | Type | Build ID | Build machine | Time to build |
1 | Linux | 20110803131630 | mv-moz2-linux-ix-slave04 | 37 mins, 8 secs |
Mac | bm-xserve11 | 2 hrs, 2 mins, 58 secs | ||
Windows | mw32-ix-slave03 | 1 hrs, 42 mins, 22 secs |
Notes
Build 1
Done on buildbot-master08.
- Marked "Any master", "mozilla-1.9.2", "Any builder" for a clobberer
- Reserved 7 slaves:
echo 7 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1
- Landed release config changes
- Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_6_20_{BUILD1,RELEASE}
- Updated checkouts on bm08, ran checkconfig
- Reconfiged
- Ran release_sanity.py, which passed:
cd /builds/buildbot/build1/master source ../bin/activate PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.20 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001
- Started the release:
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.20 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py localhost:9001
E-mail Metrics
Sent the following mail to metrics@mozilla.com:
Firefox,firefox,firefox,3.6.20,3.6 Firefox,firefox,firefox,3.6.21pre,3.6
Tag
No problems
Start autosign
Once mozilla-1.9.2 was tagged, on keymaster, started signcodepwd.exe, opened a cygwin terminal and ran the following:
PRODUCT=firefox VERSION=3.6.20 BUILD=1 TAG=FIREFOX_3_6_20_RELEASE REPO='releases/mozilla-1.9.2' EMAIL=release@mozilla.com KEYDIR=d:/2010-keys RELCONFIG=mozilla/release-firefox-mozilla-1.9.2.py cd ${HOME}/hg-tools/ hg pull && hg up -r ${TAG} mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . rsync -av ~/hg-tools/lib/python/ ./lib/python/ make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}
Bouncer Submitter
No problems
Source
No problems
Build
Firefox
No problems
Unittests / Talos
All unittests were green. tscroll on Fedora was orange because the browser froze. We don't run that test on regular checkins for 1.9.2, so I decided to ignore it.
XULRunner
Source
No problems
Build
No problems
Signing
Signed with:
#!/bin/bash set -x set -e PRODUCT=xulrunner VERSION=3.6.20 BUILD=1 TAG=FIREFOX_3_6_20_RELEASE REPO="releases/mozilla-1.9.2" EMAIL=release@mozilla.com KEYDIR=d:/2010-keys cd ~/hg-tools hg pull hg up -r ${TAG} mkdir -p ~/signing-work/${PRODUCT}-${VERSION} cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \ BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \ --exclude=*.txt --exclude=*-symbols.zip \ xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \ unsigned-build${BUILD} rsync -av --exclude=unsigned unsigned-build${BUILD}/ \ signed-build${BUILD}/ rsync -av unsigned-build${BUILD}/unsigned/ \ signed-build${BUILD}/ make checksum-files create-sigs stage verify-sigs \ PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \ REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 rmdir signed-build${BUILD}/contrib{,-localized} rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \ signed-build${BUILD}/ \ xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/
Repacks
Firefox
No problems
Firefox Signing
No problems
L10N verify
Didn't fire due to bug 676013. Reconfiged to pick up that bug (it had already been moved into production), then used force build to retrigger. No locale differences, as expected.
Updates
No problems
Update verify
A couple of chunks failed due to bug 664539. Retriggered them and they passed. Didn't inspect the logs further.
3.6.18 -> 3.6.20 Partials
Because most of our Windows and Linux users are on 3.6.18 still, we need to do a manual updates run to generate partial MARs for them, and then merge the snippets together. Done on linux-ix-slave15, because it already had 3.6.20 MARs on it:
# cltbld@aus2-staging # Make copies of the existing snippet dirs, because we'll be uploading to them cd /opt/aus2/snippets/staging rsync -av Firefox-3.6.20-build1-test/ Firefox-3.6.20-build1-test-pre-3.6.18-partials/ rsync -av Firefox-3.6.20-build1-beta/ Firefox-3.6.20-build1-beta-pre-3.6.18-partials/ rsync -av Firefox-3.6.20-build1/ Firefox-3.6.20-build1-pre-3.6.18-partials/
# cltbld@linux-ix-slave15 /builds/slave/rel-m-192-updates/build/patcher-configs wget --no-check-certificate -Omoz192-branch-from-3.6.18-patcher2.cfg "https://bugzilla.mozilla.org/attachment.cgi?id=550727" cd .. perl patcher2.pl --download --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-from-3.6.18-patcher2.cfg perl patcher2.pl --create-patches --partial-patchlist-file=patchlist.cfg --app=firefox --brand=Firefox --config=patcher-configs/moz192-branch-from-3.6.18-patcher2.cfg find temp/firefox/3.6.18-3.6.20/ftp/firefox/nightly/3.6.20-candidates/build1 -type f -exec chmod 644 '{}' \; find temp/firefox/3.6.19-3.6.20/ftp/firefox/nightly/3.6.20-candidates/build1 -type d -exec chmod 755 '{}' \; rsync -av -e 'ssh -oIdentityFile=~/.ssh/ffxbld_dsa' '--exclude=*complete.mar' temp/firefox/3.6.18-3.6.20/ftp/firefox/nightly/3.6.20-candidates/build1/update/ ffxbld@stage-old.mozilla.org:/pub/mozilla.org/firefox/nightly/3.6.20-candidates/build1/update/ rsync --include=3.6.18 --exclude=3.6* -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' temp/firefox/3.6.18-3.6.20/aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.20-build1-test/ rsync --include=3.6.18 --exclude=3.6* -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' temp/firefox/3.6.18-3.6.20/aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.20-build1-beta/ rsync --include=3.6.18 --exclude=3.6* -av -e 'ssh -oIdentityFile=~/.ssh/cltbld_dsa' temp/firefox/3.6.18-3.6.20/aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/Firefox-3.6.20-build1/
# cltbld@aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip Firefox-3.6.20-build1-test ~/bin/pushsnip Firefox-3.6.20-build1-test
As expected, the output from pushsnip only mentioned 3.6.18 snippets being overwritten.
Update Verify
Retriggered chunks 1 & 6 of update verify for each platform (chunk #6 contains full tests for de, en-US and ru; chunk #1 contains quick tests for all other locales). All green. Looked over a couple for sanity, and they appeared to be doing the right thing (downloading & applying 3.6.18-3.6.20 MARs for partials, using complete MARs for completes).
Add Bouncer Entries
Because this is a set of differently named files, we need special Bouncer entries for them. Manually created "Firefox-3.6.20-Partial-3.6.18" and added locations for each OS to it.
Major Update
Used "Force Build" to start the "release-mozilla-1.9.2-major_update" builder.
- First attempt failed because the build directory already existed (not sure why clobberer didn't get rid of it....).
- Marked "release-mozilla-1.9.2-major_update" for a clobberer and tried force build again.
- Second attempt still didn't clobber on the second time (also had issues because I didn't back out the update verify changes).
- Filed bug 676534 and clobbered mv-moz2-linux-ix-slave05:/builds/slave/rel-m-192-mu manually instead.
- Backed out the build/tools changes, then tried force build again.
No problems on the third attempt.
Update verify
No updates found for as, ka, or oc, because they were dropped between 3.6 and 5.0. No problems with any other locales, didn't look over the logs in further detail.
Socorro Product Data Updates
Added 3.6.20 and 3.6.21pre with the crash stats admin page
Check permissions / AV scan
No viruses or permissions issues.
Triggered a second time after creating 3.6.18 partials
Reset reserved slaves
echo 0 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1
Push to beta
Ahead of time, ran backupsnip:
# cltbld@aus2-staging ~/bin/backupsnip Firefox-3.6.20-build1-beta ~/bin/pushsnip Firefox-3.6.20-build1-beta
I should've done this right away, but on August 12 I unthrottled the beta channel with this AUS config update.
Push to internal mirrors
Pushed to internal mirrors on Monday morning (~8am PDT) to hopefully speed up the external push. Added '- firefox/releases/3.6.20' to /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude on surf. After waiting 10 minutes, started the push to mirrors builder with the following properties:
release_config: mozilla/release-firefox-mozilla-1.9.2.py script_repo_revision: FIREFOX_3_6_20_RELEASE
Push XULRunner to Mirrors
# xrbld@stage rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip \ /pub/mozilla.org/xulrunner/nightly/3.6.20-candidates/build1/ \ /pub/mozilla.org/xulrunner/releases/3.6.20/
Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added
- xulrunner/releases/1.9.2.19
...to exclude the previous release. </pre>
Update XULRunner wiki page
Updated XULRunner wiki pages:
https://developer.mozilla.org/en/XULRunner_1.9.2_Release_Notes
https://developer.mozilla.org/en/Gecko_SDK
Push to external mirrors
Because the files have already been copied to the releases directory, I removed the following line from /pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude to start the push:
- firefox/releases/3.6.20
Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to carry 3.6.20 instead of 3.6.19.
Final verification
- Win32 was green
- Mac was red - there were three '503 Service Temporarily Available' errors from 3347-mozilla.voxcdn.com. Lots of other good requests afterwards, so ignored
- Linux was red - another 503, and four 404's from www.mirrorservice.org. Disabled the latter until they finish syncing up all their nodes, and commented on bug 632283
Push to Release Channel
Legneato would like to push the MU, even though it's 3.6.20 --> 5.0.1, so that people still have a path if they go Help > Check for Updates after getting 3.6.20. QA is pending on that path, but throttling is already in place. About a week after 3.6.20 we'll do 3.6.20 --> 6.0.1, but dependent on billboards from bug 668108.
# cltbdl@aus2-staging ~/bin/backupsnip Firefox-3.6.20-build1 ~/bin/pushsnip Firefox-3.6.20-build1
Update Symlinks
# ffxbld@stage cd /pub/mozilla.org/firefox/releases rm latest-3.6 && ln -s 3.6.20 latest-3.6
Push 3.6.20 -> 5.0.1 MUs
We didn't push these on release day, but after the fact did:
# cltbld@aus2-staging ~/bin/backupsnip Firefox-3.6.20-build1-5.0.1-build1-MU ~/bin/pushsnip Firefox-3.6.20-build1-5.0.1-build1-MU ~/bin/pushsnip Firefox-3.6.20-build1-5.0.1-build1-MU-beta
3.6.20 -> 6.0 MU
Generation
This was done well after the original release because release notes were not ready at release time bug 668108.
- Landed release config update & reconfiged master
- Used "Force Build" to start major update automation
- Killed it with "stop build" after noticing that the patcher/update verify config bump steps were missing
- bm08 had a local patch disabling these, probably leftover from 7.0b1
- Backed out the local changes, reconfiged
- Used "force build" again
- Automation completed successfully, but detailsURL properties of snippets didn't have the locale substituted in.
- This is because we the URL "http://www.mozilla.com/%LOCALE%/firefox/latest/details/from-3.6.html" was used - and patcher requires %LOCALE% to be all lowercase.
To fix, did the following:
# cltbld@aus2-staging cd /opt/aus2/snippets/staging rsync -av Firefox-3.6.20-build1-6.0-build1-MU-test/ Firefox-3.6.20-build1-6.0-build1-MU-test.bak/ for snip in `find Firefox-3.6.20-build1-6.0-build1-MU-test -type f`; do locale=`echo $snip | cut -f6 -d'/'`; sed -i -e "s/%LOCALE%/$locale/" $snip; done rsync -av Firefox-3.6.20-build1-6.0-build1-MU-beta/ Firefox-3.6.20-build1-6.0-build1-MU-beta.bak/ for snip in `find Firefox-3.6.20-build1-6.0-build1-MU-beta -type f`; do locale=`echo $snip | cut -f6 -d'/'`; sed -i -e "s/%LOCALE%/$locale/" $snip; done rsync -av Firefox-3.6.20-build1-6.0-build1-MU/ Firefox-3.6.20-build1-6.0-build1-MU.bak/ for snip in `find Firefox-3.6.20-build1-6.0-build1-MU -type f`; do locale=`echo $snip | cut -f6 -d'/'`; sed -i -e "s/%LOCALE%/$locale/" $snip; done # should have no output grep -R '%LOCALE%' Firefox-3.6.20-build1-6.0-build1-MU* | grep -v '.bak' # re-push test snippets ~/bin/pushsnip Firefox-3.6.20-build1-6.0-build1-MU-test
Update Verify
All green, didn't look at the logs in detail.