Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

bhearsum, nthomas
Tracking bug

Signed-off Revision(s)

Build1: 7bcfe160efec

Tags

Build #	Tag	Changeset
1	GECKO1929_20100824_RELBRANCH	7bcfe160efec
	FIREFOX_3_6_9_BUILD1 FIREFOX_3_6_9_RELEASE	340d4a429fe6

Build data

Build #	Type	Build ID	Build machine	Time to build
1	Linux	20100824144629	mv-moz2-linux-ix-slave12	en-US: 42mins repack: 30mins (4 ix slaves)
	Mac	20100824144458	bm-xserve19	en-US: 1hr 37mins repack: 40mins (1 xserve, 7 minis)
	Windows	20100824153629	mw32-ix-slave02	en-US: 1hr 48mins repack: 1hr 14min (3 ix slaves)

Build 1

• Done on pm.
• Marked pm for a clobber, all builders, all branches with clobberer
• Moved the following to pm:
  • mv-moz2-linux-ix-slave09,12-14, moz2-linux-slave{30,31,32,33}
  • mw32-ix-slave{02,04,05,08}
  • bm-xserve19, moz2-darwin9-slave{09,10,11,12,21,22,23}
  • After moving the above slaves marked the following for clobbering again (because I'm not 100% sure of the behaviour of clobberer after a slave moves masters):
    • linux_{repack,update_verify,l10n_verification}
    • macosx_{repack,update_verify,l10n_verification}
    • updates
• Landed automation config
• Deleted .pyc files

# cltbld@production-master
rm /builds/buildbot/moz2-master/*.pyc

• Reconfiged master
• Closed Firefox3.6 tree
• Started automation

buildbot sendchange --username=bhearsum --master=localhost:9010 --branch=releases/mozilla-1.9.2 -m "Firefox 3.6.9 build1" Go

Tag

No problems. Manually tagged buildbot-configs, buildbotcustom, and tools with FIREFOX_3_6_9_{BUILD1,RELEASE}. Reopened tree and landed version bump.

Source

No problems with Firefox source. XULRunner source timed out when cloning mozilla-1.9.2. hg server was heavily loaded at the time; manually clobbered the slave and kicked a new run.

Build/Repack

• Linux: no problems with the en-US or l10n builds
• Mac: no problems with en-US or l10n builds
• Windows: no problems with en-US or l10n builds

XULRunner Builds

No problems

XULRunner Sign

Brought forward from 3.6.8 notes.

#!/bin/bash
set -x
set -e

PRODUCT=xulrunner
VERSION=1.9.2.9
BUILD=1
TAG=FIREFOX_3_6_9_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=nthomas@mozilla.com
KEYDIR=d:/2009-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Partner Repacks

Done by the automation but not useful until bug 554321 is resolved.

Unittests

sendchange worked fine for unit tests, running on builders as expected. Didn't get any talos runs bug 590369.

Results

• Linux: PASS, all green
• Mac: PASS, all green
• Windows: CONDITIONAL PASS, mochitest hit known orange bug 557432

41501 ERROR TEST-UNEXPECTED-FAIL | /tests/content/media/test/test_playback.html | Test timed out.

Return slaves

Gave back

• mv-moz2-linux-ix-slave09,12-14, moz2-linux-slave{30,31,32}
• mw32-ix-slave{02,04,05,08}
• bm-xserve19, moz2-darwin9-slave{09,10,11,12,21,22}

Kept

• moz2-linux-slave33
• moz2-darwin9-slave23

Sign

Removed en-US.xpi from stage.

Instructions

#!/bin/bash
set -x
set -e

PRODUCT=firefox
VERSION=3.6.9
BUILD=1
TAG=FIREFOX_3_6_9_RELEASE
REPO="releases/mozilla-1.9.2"
EMAIL=nthomas@mozilla.com
KEYDIR=d:/2009-keys

cd ~/hg-tools
hg pull
hg up -r ${TAG}

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} \
  KEYDIR=${KEYDIR} USE_NEW=1

Reviewed stdout, win32_signing_build1.log.

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} \
    BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Got "Signature verification succeeded" email.

L10N Verify

No problems. New locales ast, gd; long diffs due to lt updating their dictionaries; no binary differences.

Updates

3.6* -> 3.6.9

Generation

No problems.

Update Verify

Automated builds ran green for linux and mac. Windows had two failures (and generally took forever):

Using  https://aus2.mozilla.org/update/1/Firefox/3.6b2/20091108181924/WINNT_x86-msvc/ta/betatest/update.xml?force=1
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.6b2/20091108181924/WINNT_x86-msvc/ta/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
...
Using  https://aus2.mozilla.org/update/1/Firefox/3.6b1/20091029171059/WINNT_x86-msvc/nb-NO/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.6b1/20091029171059/WINNT_x86-msvc/nb-NO/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

which is probably stage being unhelpful. Both AUS urls have sensible output, and the hash & file size match the file on stage.

Posted patch to test 3.6.7 as thoroughly as everything else, should rerun after landing that.

3.0.19 -> 3.6.9 MU

Generation

on moz2-linux-slave33

mkdir -p /builds/slave/3.0.19-3.6.9-major-update/snippets
cd /builds/slave/3.0.19-3.6.9-major-update/snippets

cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:cltbld@cvs.mozilla.org:/cvsroot co -r UPDATE_PACKAGING_R9 -d MozBuild mozilla/tools/release/MozBuild

export CVSROOT=:ext:cltbld@cvs.mozilla.org:/cvsroot
# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R9 --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.6.9 3.0.19-3.6.9
cd ../..

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz19-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Quick Verification

cd /builds/slave/3.0.19-3.6.9-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.9/

# releasetest == beta
find aus2.test -type d -iregex '.*/releasetest$' | perl -nle '$a = $_; $a =~ s/releasetest/beta/; $a =~ s/aus2\.test/aus2.beta/; system("diff -r -u $_ $a");'
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/beta$/releasetest/; $a =~ s/aus2\.beta/aus2.test/; system("diff -r -u $_ $a");'

# beta == release
find aus2.beta -type d -iregex '.*/beta$' | perl -nle '$a = $_; $a =~ s/aus2\.beta/aus2/; $a =~ s/beta/release/; system("diff -r -u $_ $a");'
find aus2 -type d -iregex '.*/release$' | perl -nle '$a = $_; $a =~ s/release$/beta/; $a =~ s/aus2/aus2.beta/; system("diff -r -u $_ $a");'

Push to AUS

cd /builds/slave/3.0.19-3.6.9-major-update/snippets/patcher/temp/firefox/3.0.19-3.6.9
rsync -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20100825-Firefox-3.0.19-3.6.9-MU-test
rsync -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20100825-Firefox-3.0.19-3.6.9-MU-beta
rsync -av aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20100825-Firefox-3.0.19-3.6.9-MU

Enable test snippets

~/bin/backupsnip 20100825-Firefox-3.0.19-3.6.9-MU-test
~/bin/pushsnip 20100825-Firefox-3.0.19-3.6.9-MU-test

Verify

On moz2-linux-slave33, mw32-ix-slave20, bm-xserve22

mkdir -p /builds/slave/verify/firefox-3019-369-major/
cd /builds/slave/verify/firefox-3019-369-major/
hg clone http://hg.mozilla.org/build/tools
 
cd tools/release/updates
platform=win32   # or mac or win32
./verify.sh -c moz19-firefox-$platform-major.cfg 2>&1 | tee $platform.log

Results

Checked the logs carefully and found the same differences documented for 3.0.19 -> 3.6's major update.

Update Bouncer

• Hit "Force Build" on bouncer_submitter builder, got an error:

Adding product: Firefox-3.6.9
Locales: af, ar, as, ast, be, bg, bn-BD, bn-IN, ca, cs, cy, da, de, el, en-GB, en-US, eo, es-AR, es-CL, es-ES, es-MX, et, eu, fa, fi, fr, fy-NL, ga-IE, gd, gl, gu-IN, he, hi-IN, hr, hu, id, is, it, ja, ja-JP-mac, ka, kk, kn, ko, ku, lt, lv, mk, ml, mr, nb-NO, nl, nn-NO, oc, or, pa-IN, pl, pt-BR, pt-PT, rm, ro, ru, si, sk, sl, sq, sr, sv-SE, ta, ta-LK, te, th, tr, uk, vi, zh-CN, zh-TW
FAILED: Tuxedo API error. URL: https://bounceradmin.mozilla.com/api/product_add/
POST data: product=Firefox-3.6.9&languages=af&languages=ar&languages=as&languages=ast&languages=be&languages=bg&languages=bn-BD&languages=bn-IN&languages=ca&languages=cs&languages=cy&languages=da&languages=de&languages=el&languages=en-GB&languages=en-US&languages=eo&languages=es-AR&languages=es-CL&languages=es-ES&languages=es-MX&languages=et&languages=eu&languages=fa&languages=fi&languages=fr&languages=fy-NL&languages=ga-IE&languages=gd&languages=gl&languages=gu-IN&languages=he&languages=hi-IN&languages=hr&languages=hu&languages=id&languages=is&languages=it&languages=ja&languages=ja-JP-mac&languages=ka&languages=kk&languages=kn&languages=ko&languages=ku&languages=lt&languages=lv&languages=mk&languages=ml&languages=mr&languages=nb-NO&languages=nl&languages=nn-NO&languages=oc&languages=or&languages=pa-IN&languages=pl&languages=pt-BR&languages=pt-PT&languages=rm&languages=ro&languages=ru&languages=si&languages=sk&languages=sl&languages=sq&languages=sr&languages=sv-SE&languages=ta&languages=ta-LK&languages=te&languages=th&languages=tr&languages=uk&languages=vi&languages=zh-CN&languages=zh-TW
Traceback (most recent call last):
  File "tuxedo-add.py", line 143, in tuxedoRequest
    return urllib2.urlopen(request).read()
  File "/tools/python-2.5.1/lib/python2.5/urllib2.py", line 121, in urlopen
    return _opener.open(url, data)
  File "/tools/python-2.5.1/lib/python2.5/urllib2.py", line 380, in open
    response = meth(req, response)
  File "/tools/python-2.5.1/lib/python2.5/urllib2.py", line 491, in http_response
    'http', request, response, code, msg, hdrs)
  File "/tools/python-2.5.1/lib/python2.5/urllib2.py", line 418, in error
    return self._call_chain(*args)
  File "/tools/python-2.5.1/lib/python2.5/urllib2.py", line 353, in _call_chain
    result = func(*args)
  File "/tools/python-2.5.1/lib/python2.5/urllib2.py", line 499, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 400: BAD REQUEST

• • Wenzel said this was caused by new locales 'ast' and 'gd' not being in Bouncer's configuration. After the locales were added to it this was fixed. bug 590902

Push snippets to beta channel

~/bin/backupsnip 20100824-Firefox-3.6.9-beta
~/bin/pushsnip 20100824-Firefox-3.6.9-beta

EU Ballot Build

PRODUCT=firefox
VERSION=3.6.9
BUILD=1
TAG=FIREFOX_3_6_9_RELEASE
REPO='releases/mozilla-1.9.2'
EMAIL=bhearsum@mozilla.com
KEYDIR=d:/2009-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
#start signcode
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}-repacks/partners
cd ~/signing-work/${PRODUCT}-${VERSION}-repacks/partners
cp ~/hg-tools/release/signing/* .
mkdir unsigned
scp cltbld@surf.mozilla.org:/tmp/euballot-369-unsigned.tar .
cd unsigned
tar xvf ../euballot-369-unsigned.tar
cd ..
make PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} TAG=${TAG} REPO=${REPO} EMAIL=${EMAIL} USE_NEW=1 setup
# This next step will fail to find the win32 log and/or xpis.
# This is expected.
cd unsigned/win32
../../signing/sign-release ${KEYDIR}
../../signing/sign-files .
cd ../..
mv unsigned signed
cd signed
tar -vcf ../euballot-369-signed.tar .
cd ..
scp euballot-369-signed.tar cltbld@stage.mozilla.org:/tmp

Staged with:

# ffxbld@stage
# get copy of signed tarball to /tmp/
mkdir /tmp/ffxbld
cd /tmp/ffxbld
tar xfv ../euballot-369-signed.tar
mv win32 win32-EUballot
mv -v win32-EUballot /pub/mozilla.org/firefox/nightly/3.6.9-candidates/build1/

Final checks before push

Then do the normal checks

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.6.9-candidates/build1/
find . ! -user ffxbld | grep -v 'contrib*'
find . ! -group firefox | grep -v 'contrib*'
find . -type f ! -perm 644 | grep -v 'unsigned*'
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*' | grep -v unsigned
find . -maxdepth 1 -type d ! -perm 2775 -name 'contrib*'

Files and dirs in unsigned/partner-repacks have 755 and 775 permissions. We're not going to push those files.

Virus check

• clamdscan does not work. clamd was not running
  • 588252 - Ensure that clamd is running on surf
• I started this too late and slowed down the push to mirrors (bug 594092)
• We excluded purposely "unsigned" from the command to speed things up.

#ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.6.9-candidates/build1
clamscan . && clamscan -r contrib* linux-i686 mac win32 source update win32-EUballot
----------- SCAN SUMMARY -----------
Known viruses: 824443
Engine version: 0.96.1
Scanned directories: 1
Scanned files: 9
Infected files: 0
Data scanned: 0.93 MB
Data read: 0.47 MB (ratio 1.98:1)
Time: 7.518 sec (0 m 7 s)
----------- SCAN SUMMARY -----------
Known viruses: 824443
Engine version: 0.96.1
Scanned directories: 505
Scanned files: 1236
Infected files: 0
Data scanned: 9166.50 MB
Data read: 7189.69 MB (ratio 1.27:1)
Time: 4212.303 sec (70 m 12 s)

Push to mirrors

After GO from release-drivers.

#ffxbld@stage
rsync -av --exclude=*.log --exclude=*.txt --exclude=*unsigned*   --exclude=*.crashreporter-symbols.zip --exclude=*.tests.tar.bz2*   --exclude=*partner-repacks*   /pub/mozilla.org/firefox/nightly/3.6.9-candidates/build1/   /pub/mozilla.org/firefox/releases/3.6.9/

#cltbld@stage
vim /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude
# replace 3.6.9 with 3.6.8

Push Updates to release/beta channels

3.6.x to 3.6.9 and 3.0.19 -> 3.6.9

#cltbld @aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20100824-Firefox-3.6.9
~/bin/backupsnip 20100825-Firefox-3.0.19-3.6.9-MU

~/bin/pushsnip 20100824-Firefox-3.6.9
~/bin/pushsnip 20100825-Firefox-3.0.19-3.6.9-MU-beta
~/bin/pushsnip 20100825-Firefox-3.0.19-3.6.9-MU

Final verify

• I downloaded the final_verification.log and I used this:

grep HTTP/1.0 text | grep -v 302 | grep -v 200

• No problems

Release

As ffxbld@stage:

cd /home/ftp/pub/firefox/releases
rm latest-3.6 && ln -s 3.6.9 latest-3.6

Sent the following mail to metrics@mozilla.com:

Firefox,firefox,firefox,3.6.9,3.6
Firefox,firefox,firefox,3.6.10pre,3.6

Push EU Ballot build

Added bouncer entries. Filed bug 595192 to have the browser choice website updated.

Push XulRunner to mirrors

# xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned*  --exclude=*-symbols.zip \
  /pub/mozilla.org/xulrunner/nightly/1.9.2.9-candidates/build1/ \
  /pub/mozilla.org/xulrunner/releases/1.9.2.9/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.2.8

Update XulRunner wiki

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!

• Release Notes
• Gecko_SDK