Releases/Firefox 3.5.17/BuildNotes
Contents
- 1 Notes About Releasing
- 2 Build Engineers
- 3 Bugs hit
- 4 Signed-off Revision(s)
- 5 L10N changesets
- 6 Tags
- 7 Build data
- 8 Notes
- 8.1 Build 1
- 8.1.1 Tag
- 8.1.2 Source
- 8.1.3 Build
- 8.1.4 L10n Repack
- 8.1.5 XULRunner Source & Build
- 8.1.6 Sign
- 8.1.7 L10n Verify
- 8.1.8 Updates
- 8.1.9 Update Verify
- 8.1.10 Socorro product data updates
- 8.1.11 Update Bouncer
- 8.1.12 Major Update Generation (-> 3.6.14)
- 8.1.13 Publish updates to beta channel
- 8.1.14 Final checks before push
- 8.1.15 Virus check
- 8.1.16 Major Update Generation (-> 3.6.14 build2)
- 8.1.17 Major Update Generation (-> 3.6.14 build3)
- 8.1.18 Pre-push Checks
- 8.1.19 Push to mirrors
- 8.1.20 Final verify
- 8.1.21 Push updates to release channel
- 8.1.22 Push 3.5.17 -> 3.6.14 MU
- 8.1.23 Release
- 8.1.24 XULRunner
- 8.1.25 MU to 3.6.17
- 8.1 Build 1
Notes About Releasing
Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed
Build Engineers
bhearsum/coop
bug 624494
Bugs hit
- bug 626164 - Clobber won't work for build1
- bug 637886 - throttling patch not ready ahead of time
- bug 629256 - we disabled MU for broken locales for build 1, but then didn't do the same for build 2 and build 3. Needed to push out the unmodified MU list due to time constraints, so broken locales will continue to be broken.
Signed-off Revision(s)
Build 1: bc91b067f42f
L10N changesets
Updates l10n-changesets obtained from the l10n shipping dashboard.
Tags
Build # | Tag | Changeset |
1 | FIREFOX_3_5_17_RELEASE FIREFOX_3_5_17_BUILD1 | bc91b067f42f |
Build data
Build # | Type | Build ID | Build machine | Time to build |
1 | Linux | 20110121161846 | linux-ix-slave14 | 29 mins, 15 secs |
Mac | 20110121150727 | bm-xserve11 | 1 hrs, 41 mins, 34 secs | |
Windows | 20110121150729 | mw32-ix-slave02 | 1 hrs, 18 mins, 25 secs |
Notes
Done on pm01
Build 1
- Set reserved slaves to 4:
# cltbld@pm01 echo 4 > /builds/buildbot/builder_master1/reserved_slaves_pm01
- Clobber won't work for build1 because of bug 626164
- Looked around all of the fast slaves and didn't see anything lying around, so they were clobbered at some point already.
- Landed updated automation config
- Manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_17_RELEASE_FIREFOX_3_5_17_BUILD1
- Updated buildbot-configs/buildbotcustom
- Ran release_sanity.py w/ the dry run option:
PYTHONPATH=.:tools/lib/python/ python release_sanity.py -u bhearsum -V 3.5.17 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py --dryrun localhost:9010
- After it passed, ran it without --dry-run to start the release process:
PYTHONPATH=.:tools/lib/python/ python release_sanity.py -u bhearsum -V 3.5.17 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py localhost:9010
Tag
No problems
Source
No problems
Build
- Linux failed to upload symbols
- Clobbered (manually) and re-kicked with "Force Build"
- Linux failed to upload symbols again
- Catlee pointed out that the host key verification was failing
- Logged onto the slave, manually clobbered, fixed the hostkey with:
# cltbld@linux-ix-slave14 ssh -oBatchMode=no dm-symbolpush01.mozilla.org # Typed "yes" when prompted
- Started a new build with "Force Build"
- No problems with Mac or Windows
L10n Repack
- No problems on Mac or Windows
- Had to use "Force Build" with the following properties to start the Linux ones, because of the Linux en-US issues:
release_config: mozilla/release-firefox-mozilla-1.9.1.py script_repo_revision: FIREFOX_3_5_17_RELEASE
XULRunner Source & Build
No problems
Sign
Removed en-US.xpi from candidates directory
Ran the following on the signing machine
PRODUCT=firefox VERSION=3.5.17 BUILD=1 TAG=FIREFOX_3_5_17_RELEASE REPO='releases/mozilla-1.9.1' EMAIL=bhearsum@mozilla.com KEYDIR=d:/2010-keys cd ~/hg-tools hg pull && hg up -r ${TAG} mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . rsync -av ~/hg-tools/lib/python/ ./lib/python/ make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
And then to upload:
make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}
Signature verification succeeded.
L10n Verify
- Differences for all locales/platforms because of the shortened build directories.
Updates
- First attempt failed with an ssh key error, on the same slave that had key issues during the Linux build.
- Replaced keys on it with ones from a known good slave:
# cltbld@linux-ix-slave14 su -c 'umount /home/cltbld/.ssh' # multiple times, until fully unmounted rm -rf .ssh scp -r cltbld@mv-moz2-linux-ix-slave08:.ssh .
- Restarted updates with "Force Build"
- No problems
- the updates builder was triggered a third time, but not sure by what/who. It failed early, and should be harmless.
Update Verify
Triggered by hand.
- Linux:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5\ b99/20090605144815/Linux_x86-gcc3/ga-IE/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1 Using https://aus2.mozilla.org/update/1/Firefox/3.5b99/20090605144815/Linux_x8\ 6-gcc3/ga-IE/betatest/update.xml?force=1 FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.\ 5b99/20090605144815/Linux_x86-gcc3/ga-IE/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1
- Mac:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5\ .1/20090715083437/Darwin_Universal-gcc3/de/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1 Using https://aus2.mozilla.org/update/1/Firefox/3.5.1/20090715083437/Darwin_Un\ iversal-gcc3/de/betatest/update.xml?force=1 2011-01-24 15:55:38 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/ni\ ghtly/3.5.17-candidates/build1/update/mac/de/firefox-3.5.17.complete.mar [17980\ 405/17980405] -> "update/complete.mar" [1] FAIL: partial updates are larger than complete updates
- Win32:
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.\ 5.10/20100504093643/WINNT_x86-msvc/pt-BR/betatest/update.xml?force=1 FAIL: download_mars returned non-zero exit code: 1
Socorro product data updates
Added '3.5.17' and '3.5.18pre' to crash-stats with this admin page.
Update Bouncer
Added by forcing bouncer_submitter builder
Major Update Generation (-> 3.6.14)
Used 'force build' to start the major_update builder.
Had more problems with linux-ix-slave14. It turns out that this slave was still syncing with staging-puppet, which is why we were having ssh key issues.
I fixed up the slave to sync with production-puppet and the problem went away.
Major Update Verification
- mn fails for all platforms, normal and expected.
- sl blurts the following, normal and expected:
Only in source/firefox/searchplugins: iskalnik-gov-si.xml Only in source/firefox/searchplugins: najdi-si.xml Only in source/firefox/searchplugins: yahoo.xml
- Linux:
- some path differences:
- ru:
Only in source/firefox/searchplugins: rambler.xml
- Mac: the usual mac differences:
diff -r source/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js target/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js 1c1 < //@line 2 "/builds/moz2_slave/release-mozilla-1.9.1-macosx_build/build/browser/app/profile/channel-prefs.js" --- > //@line 2 "/builds/slave/release-mozilla-1.9.2-macosx_build/build/browser/app/profile/channel-prefs.js" Only in source/Firefox.app/Contents: Plug-Ins source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib
- Windows: the usual .chk file differences:
Only in source/bin: .autoreg diff -r source/bin/defaults/pref/channel-prefs.js target/bin/defaults/pref/channel-prefs.js 1c1 < //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.1-win32_build\build\browser\app\profile\channel-prefs.js" --- > //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.2-win32_build\build\browser\app\profile\channel-prefs.js" Files source/bin/freebl3.chk and target/bin/freebl3.chk differ Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ Only in source/bin: removed-files Files source/bin/softokn3.chk and target/bin/softokn3.chk differ
Publish updates to beta channel
# cltbld @ aus2-staging cd /opt/aus2/snippets/staging time ~/bin/backupsnip 20110121-Firefox-3.5.17-beta real 35m9.724s user 0m2.357s sys 0m23.525s time ~/bin/pushsnip 20110121-Firefox-3.5.17-beta real 29m36.768s user 0m2.182s sys 0m34.530s
Final checks before push
# ffxbld@stage cd /pub/mozilla.org/firefox/nightly/3.5.17-candidates/build1/ find . ! -user ffxbld # (anything in contrib/ is ok) find . ! -group firefox find . -type f ! -perm 644 find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' find . -maxdepth 1 -type d ! -perm 2775 -name 'contrib*'
contrib/solaris_pkgadd and solaris_tarball were 755 (g+s), which is OK too.
Virus check
# cltbld@stage clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.17-candidates/build1/ /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.17-candidates/build1: OK ----------- SCAN SUMMARY ----------- Infected files: 0 Time: 2694.758 sec (44 m 54 s)
Major Update Generation (-> 3.6.14 build2)
Used 'force build' to start the major_update builder.
No problems
Major Update Verification
Same "failures" as for build1.
Major Update Generation (-> 3.6.14 build3)
Used 'force build' to start the major_update builder.
No problems
Major Update Verification
Same "failures" as for build1.
Pre-push Checks
Triggered manually from waterfall.
- Revision to build: FIREFOX_3_5_17_RELEASE
- Property 1 name -> release_config: mozilla/release-firefox-mozilla-1.9.1.py
Push to mirrors
Used the new automatic builder. Failed a few times before realizing I needed to reconfig the master and re-tag the tools repo to pick up the changes (done by Rail)
No problems after that.
Final verify
A few 404s, but nothing serious.
Push updates to release channel
Please use the
- 20110126-Firefox-3.5.17-bug629256fix directory when pushing to the release channel for 3.5.x -> 3.5.17 updates
- 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix directory when pushing the major update from 3.5.17 to 3.6.14
for bug 629256. We drop updates for a few locales which have strings that make the updater crash. NThomas 23:52, 26 January 2011 (PST)
# cltbld @ aus2-staging cd /opt/aus2/snippets/staging time ~/bin/backupsnip 20110126-Firefox-3.5.17-bug629256fix time ~/bin/pushsnip 20110126-Firefox-3.5.17-bug629256fix real 17m24.216s user 0m6.373s sys 0m34.334s
Push 3.5.17 -> 3.6.14 MU
Forgot to roll a version update patch for throttling. Got that landed and then pushed in bug 637886.
# cltbld@aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix ~/bin/pushsnip 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix
But those were the 3.6.14build1 snippets, fix:
# cltbld@aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20110222-Firefox-3.5.17-3.6.14-MU ~/bin/pushsnip 20110222-Firefox-3.5.17-3.6.14-MU
Release
# ffxbld@stage cd /home/ftp/pub/firefox/releases/ rm latest-3.5 && ln -s 3.5.17 latest-3.5
Sent the following to metrics@mozilla.com
Firefox,firefox,firefox,3.5.17,3.5 Firefox,firefox,firefox,3.5.18pre,3.5
FIREFOX DONE TO HERE
XULRunner
Builds
- Linux: No problems.
- Windows: No problems.
- Mac: No problems.
Signing
Signed with:
PRODUCT=xulrunner VERSION=1.9.1.17 BUILD=1 TAG=FIREFOX_3_5_17_RELEASE REPO="releases/mozilla-1.9.1" EMAIL=coop@mozilla.com KEYDIR=d:/2010-keys mkdir -p ~/signing-work/${PRODUCT}-${VERSION} cd ~/signing-work/${PRODUCT}-${VERSION} cd ~/hg-tools hg pull hg up -r ${TAG} cd ~/signing-work/${PRODUCT}-${VERSION} cp ~/hg-tools/release/signing/* . make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \ BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \ --exclude=*.txt --exclude=*-symbols.zip \ xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \ unsigned-build${BUILD} rsync -av --exclude=unsigned unsigned-build${BUILD}/ \ signed-build${BUILD}/ rsync -av unsigned-build${BUILD}/unsigned/ \ signed-build${BUILD}/ make checksum-files create-sigs stage verify-sigs \ PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \ REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1 rmdir signed-build${BUILD}/contrib{,-localized} rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \ signed-build${BUILD}/ \ xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/
Push XULRunner to mirrors
- xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*symbols.zip \ /pub/mozilla.org/xulrunner/nightly/1.9.1.17-candidates/build1/ \ /pub/mozilla.org/xulrunner/releases/1.9.1.17/
Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added
- xulrunner/releases/1.9.1.16
...to exclude the previous release.
Update XULRunner wiki
Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!
XR DONE TO HERE
MU to 3.6.17
rail forced the MU last night.
We had to do a fast 3.6.15 release so we had to generate a new MU
# cltbld@aus2 # From any build slave cd /opt/aus2/snippets/staging ~/bin/backupsnip 20110303-Firefox-3.5.17-3.6.15-MU-beta ~/bin/backupsnip 20110303-Firefox-3.5.17-3.6.15-MU ~/bin/pushsnip 20110303-Firefox-3.5.17-3.6.15-MU-beta ~/bin/pushsnip 20110303-Firefox-3.5.17-3.6.15-MU