Contents
Notes About Releasing
Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed
Build Engineers
bhearsum - bug 638061
Signed-off Revision(s)
- Build 1: 290712e55ade
L10N changesets
Retrieved the l10n-changesets as follows:
wget -O l10n-changesets_mozilla-2.0 https://l10n-stage-sj.mozilla.org/shipping/l10n-changesets?ms=fx4.0rc1
Hit "ship it" on Firefox 4.0rc1 page (https://l10n-stage-sj.mozilla.org/shipping/milestones)
Tags
| Build # | Tag | Changeset |
| 1 | GECKO20_2011030313_RELBRANCH | 290712e55ade |
| FIREFOX_4_0rc1_BUILD1 FIREFOX_4_0rc1_RELEASE | 5f8f494a4c29 |
Build data
| Build # | Type | Build ID | Build machine | Time to build |
| 1 | Linux [compressed .tar.bz2] | 20110303140816 | linux-ix-slave19 | 57min2s |
| Linux64 [compressed .tar.bz2] | 20110303144013 | moz2-linux64-slave05 | 1h36min23s | |
| Mac [.dmg] | 20110303140758 | moz2-darwin10-slave13 | 4h27min42s | |
| Win32 [.exe/.zip] | 20110303194838 | w32-ix-slave30 | hours and hours because of manual kicking |
Build 1
We're on pm01 for this release.
Setup
Set a clobber
Set a clobber for pm01, any release, any builder with clobberer.
Sync repositories
We're building RCs out of the mozilla-2.0 and l10n-mozilla-2.0 repositories, and we're responsible for syncing the latest from mozilla-central/l10n-central to them. To do this, did the following on an idle slave, after opening the mozilla-2.0 tree:
cd /builds/slave mkdir rc1-sync cd rc1-sync wget -O- http://hg.mozilla.org/mozilla-central/raw-file/tip/browser/locales/shipped-locales | grep -v en-US | cut -f1 -d" " > locales.txt hg clone http://hg.mozilla.org/mozilla-central cd mozilla-central hg push -e 'ssh -i ~/.ssh/ffxbld_dsa' ssh://ffxbld@hg.mozilla.org/releases/mozilla-2.0 for i in `cat locales.txt`; do hg clone http://hg.mozilla.org/l10n-central/$i; done for i in `cat locales.txt`; do hg -R $i push -e 'ssh -i ~/.ssh/ffxbld_dsa' ssh://ffxbld@hg.mozilla.org/releases/l10n-mozilla-2.0/$i; done
Only a few locales had changesets pushed, because the l10n-mozilla-2.0 were cloned earlier in the day.
Reserve Slaves
On pm01, did the following, to reserve enough fast slaves for the release:
cd /builds/buildbot/builder_master1 echo 6 > reserved_slaves_pm01
Move IX machines
Because of 638309 we had to connect some SCL-based IX machines to pm01 for the duration of this release. Moved the following:
- linux-ix-slave18 - 23
- w32-ix-slave27 - 32
Land Configs
- Landed release config updates
- Also added "force_plist_reload" to mozilla-2.0's patcher config, to ensure no strangeness with that file like we had in b9 througb 12.
Tag Repositories
Tagged build/tools, buildbotcustom, and buildbot-configs with FIREFOX_4_0rc1_RELEASE and FIREFOX_4_0rc1_BUILD1
Update Master
Updated /tools/buildbotcustom/buildbotcustom, /builds/buildbot/builder_master/tools and /builds/buildbot/configs on pm01 to the latest of the production branches. Afterwards, reconfiged with the following command:
cd /builds/buildbot/builder_master1 make reconfig && tail -F twistd.log | grep configuration
Run release_sanity.py
PYTHONPATH=/builds/buildbot/builder_master1:/builds/buildbot/builder_master1/tools/lib/python python /builds/buildbot/builder_master1/tools/buildbot-helpers/release_sanity.py -u bhearsum -V 4.0rc1 --branch mozilla-2.0 --build-number 1 -c release-firefox-mozilla-2.0.py --dryrun localhost:9010
Kick it off
PYTHONPATH=/builds/buildbot/builder_master1:/builds/buildbot/builder_master1/tools/lib/python python /builds/buildbot/builder_master1/tools/buildbot-helpers/release_sanity.py -u bhearsum -V 4.0rc1 --branch mozilla-2.0 --build-number 1 -c release-firefox-mozilla-2.0.py --bypass localhost:9010
Tag
No problems.
Bouncer Update
Triggered automatically after tagging, no issues.
Start autosign
Once mozilla-2.0 finished tagging, used these instructions to start autosign
Autosign failed "Incorrect number of .asc files" due to extra index.html file on stage. Added '-not -name "*.html"' to TOTAL_FILE macro and reran "make verify fake-upload postsing ..."
Source
No problems
Build/Repack
- Linux: no problems with en-US
- L10n repacks failed the first time because I forgot to update the mozconfigs for the l10n-mozilla-2.0 repositories. They failed a second time because I forgot to retag buildbot-configs after landing the fix.
- Because of failed repacks, "linux_repack_complete" builder didn't fire. Used "Force Build" to trigger it manually.
- Linux 64-bit: no problems with en-US or repacks
- Win32
- lost connection to w32-ix-slave29 during linking, buildbot tried to retry on w32-ix-slave30 automatically, but couldn't connect to it. catlee rebooted slave30, and then updated the db so the request would be retried again. slave29 picked it up a second time.
- 2nd build on slave29 failed, automatically retried and picked up by slave30 this time. catlee moved all w32-ix slaves to buildbot-master1:9010
- hit rebuild on build#1 so we could get a build going on a VM at the same time in case the one on the IX machine dies again
- final build attempt on IX through buildbot died because bhearsum tried to trick the linker into being verbose, but caused a bad invocation instead. Kicked it through manually by doing the following:
# In MSYS shell through VNC on w32-ix-slave30 cd /e/builds/moz2_slave/rel-2.0-w32-bld/build export MOZ_OBJDIR=obj-firefox export MOZ_PKG_PRETTYNAMES=1 export MOZ_PKG_VERSION=4.0rc1 make -f client.mk profiledbuild MOZ_BUILD_DATE=20110303194838 cd obj-firefox make buildsymbols export POST_SYMBOL_UPLOAD_CMD=/usr/local/bin/post-symbol-upload.py export SYMBOL_SERVER_HOST=dm-symbolpush01.mozilla.org export SYMBOL_SERVER_PATH=/mnt/netapp/breakpad/symbols_ffx/ export SYMBOL_SERVER_SSH_KEY="/c/Documents and Settings/cltbld/.ssh/ffxbld_dsa" export SYMBOL_SERVER_USER=ffxbld make uploadsymbols make package make installer make -C obj-firefox/tools/update-packaging cd ../ python build/config/printconfigsetting.py build/obj-firefox/dist/bin/application.ini App BuildID > build/obj-firefox/dist/win32_info.txt export POST_UPLOAD_CMD="post_upload.py -p firefox -n 1 -v 4.0rc1 --release-to-candidates-dir" export UPLOAD_EXTRA_FILES=win32_info.txt export UPLOAD_HOST=stage.mozilla.org export UPLOAD_SSH_KEY=~/.ssh/ffxbld_dsa export UPLOAD_TO_TEMP=1 export UPLOAD_USER=ffxbld cd build/obj-firefox make upload
- Manually sent "unsigned win32 en-US build available" mail
- Manually forced the 6 win32 l10n builders with the following properties set:
- release_config: mozilla/release-firefox-mozilla-2.0.py
- script_repo_revision: FIREFOX_4_0rc1_RELEASE
- OSX: no problems with en-US or repacks
Added index.html the firefox/nightly/4.0rc1-candidates/build1/ to redirect the curious/obsessive ftp polling types.
Returned linux-ix-slave18 - 23 to bm04, since they had gotten hung anyway.
Unittests & Talos
None of these ran - bug 641399.
Signing
No problems, autosign worked fine. Got "Signature verification succeeded" mail.
Unreserve Slaves
On pm01, did the following:
cd /builds/buildbot/builder_master1 echo 0 > reserved_slaves_pm01
L10N Verification
post_sign bits weren't triggered. FtpPoller was polling http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/4.0rc1-candidates/build1/ which instead of showing a directory listing was redirecting to http://www.mozilla.com/en-US/firefox/comingsoon/
Forced.
Results:
- All Platforms:
- "fa" only in RC1.
- This is expected because it's a new locale as of this release.
- Big spew for 'br' locale, which had a big dictionary upgrade in bug 637423
- Small updates for various other locales
- "fa" only in RC1.
Nothing else of note, PASS.
Updates
post_sign bits weren't triggered. FtpPoller was polling http://stage-old.mozilla.org/pub/mozilla.org/firefox/nightly/4.0rc1-candidates/build1/ which instead of showing a directory listing was redirecting to http://www.mozilla.com/en-US/firefox/comingsoon/
Forced.
- Failed a few times because it couldn't free enough space
- Failed another time because it couldn't read the a buildid properly.
- Problem turned out to be that the win32_info.txt wasn't formatted correctly, which was fallout from manually forcing it through earlier. After fixing it to be in the correct format (buildid=2011xxxxxxxxxx), forced another run.
- This run succeeded
Update Verify
Manually forced update verify builders. Results:
- All platforms/locales:
- Update channel differences ("beta" vs "release")
- Test Pilot removal listed in removed-files
The update channel change, and test pilot listing in removed-files are both expected and innocuous. Nothing else of note, so this is a PASS.
Permissions Check
Triggered through force build with the following properties:
- release_config: mozilla/release-firefox-mozilla-2.0.py
- script_repo_revision: FIREFOX_4_0rc1_RELEASE
Virus Scan
Triggered through force build with the following properties:
- release_config: mozilla/release-firefox-mozilla-2.0.py
- script_repo_revision: FIREFOX_4_0rc1_RELEASE
No issues found.
XULRunner Builds
Had to land a follow-up patch to enable XULRunner builds. Triggered them through "Force Build"
- Linux 32-bit: No problems
- Linux 64-bit: No problems
- Windows: No problems
- Mac: Failed to compile. Filed bug 638921 on the issue.
- A rebuild succeeded compilation, but failed due to bug 634809
- Worked around this manually with:
- A rebuild succeeded compilation, but failed due to bug 634809
# cltbld@moz2-darwin10-slave17 cd /builds/slave/rel-2.0-xr-osx64-bld/build/obj-firefox/x86_64/dist/sdk scp -i ~/.ssh/xrbld_dsa \ xulrunner-2.0.en-US.mac-x86_64.sdk.tar.bz2 \ xrbld@stage.mozilla.org:/pub/mozilla.org/xulrunner/nightly/2.0-candidates/build1/sdk/
All files were uploaded to "2.0-candidates" instead of "2.0rc1-candidates" because of bug 639888. Manually did the following to fix:
# xrbld@stage cd /home/ftp/pub/xulrunner/nightly mv 2.0-candidates 2.0rc1-candidates
XULRunner Signing
#!/bin/bash
set -x
set -e
PRODUCT=xulrunner
VERSION=2.0rc1
BUILD=1
TAG=FIREFOX_4_0rc1_RELEASE
REPO="releases/mozilla-2.0"
EMAIL=@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull
hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
--exclude=*.txt --exclude=*-symbols.zip \
xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
signed-build${BUILD}/
make checksum-files create-sigs stage verify-sigs \
PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rmdir signed-build${BUILD}/contrib{,-localized}
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
signed-build${BUILD}/ \
xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/
Socorro product data updates
Someone else had already added "4.0". I added "4.0pre".
Push to Mirrors
Forced the "release-mozilla-2.0-push_to_mirrors" builder with the following properties:
- release_config: mozilla/release-firefox-mozilla-2.0.py
- script_repo_revision: FIREFOX_4_0rc1_RELEASE
Notify AV vendors
- Notify vendors using instructions in bug 568884#c15
- this done out of sequence as missing from build notes, and manually caught. Not sure if AV vendors will have enough time to scan *today* as push-to-mirrors already well in progress. Need to do this earlier in process, see FF3.6.14, FF3.6.15 for examples.
Notify Mirrors
Normally this is done well in advance of pushing, but I forgot to do it until well after. Sent short mail to mirrors@mozilla.org informing them that 4.0rc1 was pushed.
Push Beta Channel Snippets
# cltbld@aus2-staging cd /opt/aus2/snippets/staging ~/bin/backupsnip 20110303-Firefox-4.0rc1 ~/bin/pushsnip 20110303-Firefox-4.0rc1
Send mail to metrics
Firefox,firefox,firefox,4.0,4.0 Firefox,firefox,firefox,4.0pre,4.0
Release
# cltbld@stage.mozilla.org cd /pub/mozilla.org/zz/
Updated rsync exclude files from the above dir to exclude b12
Push XULRunner to mirrors
# xrbld@stage rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*-symbols.zip \ /pub/mozilla.org/xulrunner/nightly/2.0rc1-candidates/build1/ \ /pub/mozilla.org/xulrunner/releases/2.0rc1/
Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-{pre,}releases.exclude and added:
- xulrunner/releases/2.0b12
Update XULRunner entries on wiki
Updated the links in these two wiki pages for the version change:
Remove index.html
After release, remove the index.html:
# ffxbld@stage cd /pub/mozilla.org/firefox/releases/4.0rc1 rm index.html