Notes About Releasing

Please update the Notes Template and the Release:Primer for future releases (bug fixes, changes to automation) as needed

Bugs hit

Build Engineers

{bhearsum}, {nthomas} - Tracking bug: bug 682938

Signed-off Revision(s)

Build 1: 11882be93d0a

L10N changesets

• Firefox: link to the l10n-changesets

Tags

Build #	Branch, Tags	Changeset
1	GECKO19220_2011080310_RELBRANCH, FIREFOX_3_6_21_BUILD1 FIREFOX_3_6_21_RELEASE	11882be93d0a

Build data

Firefox

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20110830092825	mv-moz2-linux-ix-slave05	40 mins, 12 secs
	Mac		bm-xserve15	2 hrs, 1 mins, 46 secs
	Windows		mw32-ix-slave12	1 hrs, 49 mins, 30 secs

Notes

Done on bm08

Build 1

• Set reserved slaves:

# cltbld@buildbot-master08
# 8 for 3.6.21, 8 for 7.0b3
echo 16 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1

• Requested a clobber for any master, mozilla-1.9.2, any builder
• Landed release config changes
• Tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_6_21_{BUILD1,RELEASE}
• After waiting for 7.0b3 configs to be landed, updated and reconfiged bm08
• Ran release_sanity.py:

cd /builds/buildbot/build1/master
source ../bin/activate
PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.21 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py --dryrun localhost:9001

• Started the release:

PYTHONPATH=. python ../tools/buildbot-helpers/release_sanity.py -u bhearsum -p firefox -V 3.6.21 --branch mozilla-1.9.2 --build-number 1 -c release-firefox-mozilla-1.9.2.py localhost:9001

E-mail Metrics

Sent the following mail to metrics@mozilla.com:

Firefox,firefox,firefox,3.6.21,3.6
Firefox,firefox,firefox,3.6.22pre,3.6

Tag

No problems

Bouncer Submitter

No problems

Source

No problems

Start autosign

Started autosign on keymaster with the following commands:

PRODUCT=firefox
VERSION=3.6.21
BUILD=1
TAG=FIREFOX_3_6_21_RELEASE
REPO='releases/mozilla-1.9.2'
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys
RELCONFIG=mozilla/release-firefox-mozilla-1.9.2.py
cd ${HOME}/hg-tools/ # if it exists, otherwise create a fresh clone
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

Build

Firefox

No problems

Unittests / Talos

XULRunner

Source

No problems

Build

No problems

Signing

Signed with:

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=3.6.21
BUILD=1
TAG=FIREFOX_3_6_21_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=release@mozilla.com
KEYDIR=d:/2010-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Repacks

Firefox

No problems

Firefox Signing

Blew up when doing signcode signing for pa-IN and zh-TW. I think this may be related to me hitting the windows key in a VNC session while signing was in progress, because that keypress somehow makes it through to the desktop. Moved signed-build1 out of the way, and started again:

# cltsign
cd signing-work/firefox-3.6.21
mv signed-build1 signed-build1.bak
# because the autosign loop doesn't support a clobbered signed-build1 dir =\
make download PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} RELCONFIG=${RELCONFIG}

L10N verify

No problems

Updates

No problems as far as uploading snippets. Pushed to internal mirrors after mar upload so QA can test releasetest (this caused issues, see below).

Slow performance on aus2-staging (NFS?) caused retry.py to timeout the pushsnip after an hour, and rerun it. Second time appeared to complete OK. Two issues filed:

• bug 683412 - no output at all from the first attempt, no idea how far it got
• bug 683417 - first pushsnip didn't actually kill all processes

Update verify

Couple of failures:

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.20/20110803131630/Darwin_Universal-gcc3/lv/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.20/20110803131630/Darwin_Universal-gcc3/zh-TW/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

In both cases there is a partial mar specified in the xml, and the file is present on the server. We should track this down some day.

Major Update

Used "Force Build" to start the "release-mozilla-1.9.2-major_update" builder. No problems

Update verify

All green, didn't look at them in detail.

Socorro Product Data Updates

UI is gone from admin page, emailed socorro-dev about it.

Check permissions/ AV scan

Forced the AV scan before the updates builder finished (but after partials were uploaded), with

release_config: mozilla/release-firefox-mozilla-1.9.2.py
script_repo_revision: FIREFOX_3_6_21_RELEASE

Got a failure for

test ! -d /pub/mozilla.org/firefox/releases/3.6.21/

because I'd already pushed to mirrors. After that it would have done a dummy push to mirrors, which is OK to miss today. Auto triggered run was cancelled, no need to run it twice.

Got the same dir test error for Check Permissions, but the permission checks before that were all fine.

Filed bug 683448

Reset reserved slaves

echo 0 > /builds/buildbot/build1/master/reserved_slaves_bm08-build1

Push to internal mirrors

Pushed to early to speed up the external push and QA testing releasestest. Added

- firefox/releases/3.6.21

to stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude.

After waiting 10 minutes, started the push to mirrors builder with the following properties:

release_config: mozilla/release-firefox-mozilla-1.9.2.py
script_repo_revision: FIREFOX_3_6_21_RELEASE

Push XULRunner to Mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/3.6.21-candidates/build1/ \
  /pub/mozilla.org/xulrunner/releases/3.6.21/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/3.6.20

...to exclude the previous release.

Update XULRunner wiki page

not done yet because devmo seems broken Updated XULRunner wiki pages: https://developer.mozilla.org/en/XULRunner_1.9.2_Release_Notes
https://developer.mozilla.org/en/Gecko_SDK

Push to external mirrors

Because the files have already been copied to the releases directory, just need to remove the following line from stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude to start the push:

- firefox/releases/3.6.21

Updated stage:/pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to carry 3.6.21 instead of 3.6.20.

Final verification

• mac: green
• win32: got a couple of '503 Service Temporarily Unavailable' from 3347-mozilla.voxcdn.com (out of 1016 requests to them)
• linux: another one of phantom FAIL's

FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6.17/20110420140830/Linux_x86-gcc3/bg/releasetest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

Push to Release Channels

We didn't have time to backupsnip this release, which was undesirable.

~/bin/pushsnip Firefox-3.6.21-build1

Unthrottling

We had a 3.6.20 --> 5.0.1 major update on release and beta, throttled so that only manual checks find it. Wanted to unthrottle the 3.6.20 --> 3.6.21 update on release asap using attachment 556657 [edit], so remove the beta MU:

# cltbld@aus2-staging
cd /opt/aus2/incoming/3/Firefox/3.6.20
find . -depth -type d -name 'beta' -exec rm -rfv {} \;
rsync -av --delete /opt/aus2/incoming/3/Firefox/3.6.20/ \
                   ffxbld@dp-ausstage01:/opt/aus2/incoming/3/Firefox/3.6.20/
 

Landed the AUS patch and filed bug bug 683443 for deployment.

Still need to do the beta snippets (on Wednesday).

Push to beta channel

~/bin/backupsnip Firefox-3.6.21-build1-beta

Update symlinks

# ffxbld@stage
cd /pub/mozilla.org/firefox/releases
rm latest-3.6 && ln -s 3.6.21 latest-3.6

Push 3.6.21 -> 5.0.1 MUs

3.6.21 -> 6.0.1 MU

Can't be shipped until bug 668108 is fixed.

Generation

• Marked "Any master", "Any release", "release-mozilla-1.9.2-major_update" for a clobber. (both of them!)
• Landed config update.

Update Verify