Firefox 2.0.0.14:BuildNotes

Build Engineers

joduinn & nthomas & bhearsum Version/config bump bug 426307

Bonsai queries

• RC1
  • cvsroot
  • (l10n bonsai link goes here)

Tags

Firedrill based on release branch for 2.0.0.13

Module	Tag	Pull date
cvsroot/mozilla	RC1: GECKO181_20080311_RELBRANCH FIREFOX_2_0_0_14_RC1 FIREFOX_2_0_0_14_RELEASE	2008-04-04 XX:XX PDT
l10n/l10n	RC1: GECKO181_20080311_RELBRANCH FIREFOX_2_0_0_14_RC1	2008-04-04 XX:XX PDT

Updated CVS Tags devmo page.

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]	RC1:			production-pacifica-vm
[Mac compressed]	RC1:			bm-xserve05
[Linux compressed]	RC1:			production-prometheus-vm

Notes

RC1

Bootstrap Tag: RELEASE_AUTOMATION_M8_1

Setup before starting:

• Bump bootstrap.cfg & master.cfg in bug, get review, land
• Did a 'cvs up' in /builds/buildbot/Automation/bootstrap-configs
  • All files are now Up-to-date
• Did a 'cvs up' in /builds/buildbot/Automation/buildbot-configs
  • master.cfg is "Locally Modified" to contain passwords. Identical to repository version otherwise.
  • Turned off the nightly builders on production-1.8-master to make sure they don't interfere with the release. (We need to figure out how to avoid this in the future).
  • update mozilla1.8 tinderbox to say "no nightlies right now"
  • Ran 'buildbot reconfig'
• On production-1.8-master, make sure updates from previous release are present. Today, we verified the contents in /home/ftp/pub/firefox/nightly/2.0.0.13-candidates/rc1.
• update /builds/tinderbox/mozilla/tools/tinderbox to the Bootstrap tag (need to revert this before restarting nightlies)
• ensure that machines have enough resources
  • production-1.8-master
    • removed (but leave the dirs so we don't have to set them up with the right perms again)
      • /data/cltbld/thunderbird-2.0.0.13/
      • /home/ftp/pub/thunderbird/nightly/*candidates
      • /home/ftp/pub/thunderbird/nightly/2008-*
      • /home/ftp/pub/thunderbird/nightly/2008/03/*
      • /home/ftp/pub/thunderbird/nightly/2008/04/*
    • Did not remove /build/tags -- plenty of space on /builds
  • On bm-xserve05 remove the contents of /builds/verify/*
  • On production-prometheus-vm, checked /builds/updates for really old releases
  • Free space on production-1.8-master before starting: /builds - 26G, /data - 26G
  • Free space on bm-xserve05 before starting: / - 36G
  • Free space on production-prometheus-vm before starting: /builds - 28G
  • Free space on production-pacifica-vm before starting: C - 36G

• Kicked off the automation:

buildbot sendchange --username=nthomas --master=localhost:9989 -m"Firefox 2.0.0.14 RC1" release

Tag

• automated - nothing to do

Source

• automated - nothing to do

Build

• automated - nothing to do

Sign

• Signing doc
• Note: clear space on production-1.8-master so that later stage step does not run out of space - wasn't required, had nearly 8GB free after Stage (started with 28GB free on /data)
• slave on production-1.8-master had dropped, killing the first sign step as it waited for the signing log
• Some partitions on the master had gone read-only (bug 407796) and a simple reboot left the buildbot master unable to start. Ran fsck on each partition and that resolved it. Editing master.cfg to start doing work at stage

L10nVerify

• automated - nothing to do
• metadiff is OK (empty, as normal for Fx2)

Generate Updates

• automated - no problems

Publish Updates to Test Channels (betatest & releasetest)

• backupsnip took longer than the default timeout of 1 hour for a Shell() call, and got killed. Reached into the slave to add a 7200 second timeout to the call, modified master.cfg to make Push the first (uncommented) step of the updateFactory, reconfig'd the master and sent a change
• worked fine, 37 minutes to backup
• Snippet dirs have prefix 20080407-Firefox-2.0.0.14

Update Verify

• automated - no problems
• logs check out fine

Stage

• automated - no problems

Push Updates to Beta Channels

While waiting for QA to give "ok", on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
 $ time ~/bin/backupsnip 20080407-Firefox-2.0.0.14-beta
 real    39m51.184s
 user    0m23.217s
 sys     0m33.234s

After QA gives "ok" to push to beta channel, on aus2-staging do:

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update mozilla/tools/release/bin/ 
 $ cd /opt/aus2/snippets/staging
 $ time ~/bin/pushsnip 20080407-Firefox-2.0.0.14-beta
 real    2m41.935s
 user    0m0.107s
 sys     0m9.752s

Sign Installers

• Done manually using these installer-signing-instructions here
• push signed bits back to same location on stage
• complete stage-merged:

# on stage
cd /data/cltbld/firefox-2.0.0.14/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

• Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-2.0.0.14/stage-merged/
~/bin/checksum-files .

• Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

• Manually added entries to download.mozilla.org

Push to mirrors

• get the last formal "go" from QA/Dev/website/IT/release-drivers
• push the stage-merged directory to the releases area:

# on stage (as cltbld)
rsync -av /data/cltbld/firefox-2.0.0.14/stage-merged/ /home/ftp/pub/firefox/releases/2.0.0.14/

Final Verification

• Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz18-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

• Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

• Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080407-Firefox-2.0.0.14
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080407-Firefox-2.0.0.14-test/$a");'

Publish Updates to Release Channel

• In case you need to cancel/abort updates, the quickest way to do this is here
• After QA finished testing the releasetest channel, enabled release channel:

 # on aus2-staging
 # put snippets on release/live channel
 $ cd /opt/aus2/snippets/staging
 $ sudo su - cltbld
 $ ~/bin/pushsnip 20080407-Firefox-2.0.0.14

• VERIFYFIX: Had to chmod +x ~/bin/pushsnip

Release

• Move forward the "latest" and "latest-2.0" symlinks:

 cd /home/ftp/pub/firefox/releases/
 rm latest
 ln -s 2.0.0.14 latest
 rm latest-2.0
 ln -s 2.0.0.14 latest-2.0

• Update stage:/etc/rsyncd-mozilla-current.exclude

Free Software Builds

• Bump the BuildTag definition in the three tinder-config.pl, like this

• Done on production machines:
  • production-pacifica-vm (win32)
  • bm-xserve05 (macosx)
  • production-prometheus-vm (linux)

• Start builds:

cd /builds/tinderbox/Fx-Mozilla1.8-FS
./build-seamonkey.pl --once --depend --config-cvsup-dir `pwd`/tinderbox-configs/ 2>&1 | tee FIREFOX_2_0_0_14_FS.log

• Stage

# as cltbld@stage
mkdir -p ~/firefox-2.0.0.14-fs/batch1/prestage
cd ~/firefox-2.0.0.14-fs/batch1
# rsync from push dirs
rsync -av /home/ftp/pub/firefox/nightly/2008-04-16-04-mozilla1.8-fs/ ./prestage/
rsync -av /home/ftp/pub/firefox/nightly/2008-04-15-14-mozilla1.8-fs/ ./prestage/
rsync -av prestage/ prestage-trimmed/
rm -v prestage-trimmed/*.zip
rsync -av prestage-trimmed/ stage/
cd stage
~/bin/groom-files --long="2.0.0.14.fs" .
chown -R cltbld:firefox * .
find . -type f -exec chmod -v 644 {} \;
find . -type d -exec chmod -v 755 {} \;
cd ../
rsync -av stage/ /home/ftp/pub/firefox/releases/2.0.0.14/contrib/free-software/