Releases/Firefox 3.0.1:BuildNotes

Build Engineers

Bonsai queries

Build data

Type	Build ID	SHA1	Push date	Build machine
[Windows installer/zip]				fx-win32-1.9-slave2
[Mac compressed]				fx-mac-1.9-slave2
[Linux compressed]				fx-linux-1.9-slave2

Notes

Build 1

Bootstrap Tag: RELEASE_AUTOMATION_M10

Setup before starting:

Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
On the slaves:
verify that linux has 'DISPLAY=:0 xhost +' set
remove /builds/verify/firefox-3.0rc3 and firefox-3.0rc2
update tinderbox to RELEASE_AUTOMATION_M10
- For win32:

cd /e/builds/tinderbox/mozilla/tools/tinderbox
cvs up -r RELEASE_AUTOMATION_M10
for dir in fx19rel fx19l10nrel xr19rel; do 
  cd /e/$dir; 
  for f in `ls *.pl *.html INSTALL README | grep -v tinder-config.pl`; do 
    cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .;
  done
  cp -pv post-mozilla-rel.pl post-mozilla.pl
done

On the linux slave, removed:
- /builds/updates/firefox-3.0rc3
- /builds/source/*
- /data/cltbld/firefox-3.0rc3
For some reason bm-xserve11 was attached to the buildbot master.
- Disconnected it, and reconnected the proper slave (bm-xserve10/fx-mac-1.9-slave2
Space on slaves before starting:
- fx-linux-1.9-slave2: 29G on /builds
- fx-mac-1.9-slave2: 24G on /
- fx-win32-1.9-slave2: 17G on d:, 2G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)

Disabled everything after Updates::Push in master.cfg (local modification) because additional patcherconfig/update work needs to be done after Updates::execute but before updates::push.

Gulped, and kicked off automation:

buildbot sendchange --username=bhearsum --master=localhost:9989 -m"Firefox 3.0.1build1 release" gogogo

Tag

Automated, no problems.

Source

Automated, no problems

Build

TinderConfig failed with:

Step TinderConfig died: ASSERT: no replacement found for appVersion at Bootstrap/Config.pm line 364, <INFILE> line 287.

Not sure why this happened ('version' should be used when appVersion isn't present, I think. Maybe it's not when oldAppVersion is present?). Added it to the config, checked in, tagged, restarted builds.

TinderConfig failed spuriously because en-US configs were already tagged. Commented out TinderConfig from master.cfg and restarted builds.

Build failed out on Mac - /usr/bin/python was pointing to python2.3 - repointed it to /tools/python/bin/python (python 2.5). not sure how this happened, or how it worked for 3.0rc1/2/3. maybe related to above comments about bm-xserve11.

Build failed out on win32 due to disk space. Deleted some trees on e: and restarted. (10G free @ time of restart)

Repack

Mac repack was done incorrectly because of the TinderConfig problem mentioned above. Only the en-US tinder-config for mac and win32 was bumped. TinderConfig was manually re-run on win32 and mac. This was done before win32 started repacking (therefore, it did the right thing the first time), but after mac had completed its full repack. All of the mac repacks were deleted from stage, and Repack::Excute and ::Push were re-run. This produced the correct builds.
No other problems

Sign

Signing doc

L10nVerify

No changes

Generate Updates

Mac snippets (3.0rc3 -> 3.0.1)

Ran PatcherConfig & Updates::Execute to get Mac snippets.
Didn't get a 'aus2.beta' directory (beta + release channel snippets were in the same directory).
Fixed that by adding 'beta-dir' to <current-update> stanza in the patcher config
Re-ran Updates::Execute - got aus2.beta this time
Did a spot check, things looked OK.

Linux & win32 snippets (3.0rc2 -> 3.0.1)

Updated patcher config to get linux & win32 partials/snippets (from: 3.0rc2, to: 3.0.1, comment out rc3 release stanza, comment out mac from rc2, comment out 3.0rc2 to 3.0rc3 past-update line)
Commented out 'clean update area' so we don't clobber the mac snippets.
backup mac snippets:

cp -R 3.0rc3-3.0.1/ 3.0rc3-3.0.1.rc3.mac
cp -R 3.0.rc3-3.0.1/ ~/3.0rc3-3.0.1.rc3.mac

Re-ran Updates::Execute to generate linux and win32 partials/snippets
- Got the following at the bottom of the log:

Can't stat /builds/updates/firefox-3.0.1/patcher/temp/firefox/3.0rc3-3.0.1/aus2.test: No such file or directory 
at Bootstrap/Step/Updates.pm line 110
Can't stat /builds/updates/firefox-3.0.1/patcher/temp/firefox/3.0rc3-3.0.1/aus2: No such file or directory 
at Bootstrap/Step/Updates.pm line 119

- This is because we're generating snippets for a release that is NOT the 'version' as set in bootstrap.cfg. Manual spotcheck below to make sure everything is ok.
Manual spotchecks:

# in aus2.test: look for non-test channels snippets in test channel dir
find . -mindepth 6 -type d | grep -v test # expect no output
# the same thing for beta, and release channel
# in aus2.beta
find . -mindepth 6 -type d | grep -v beta # expect no output
# in aus2
find . -mindepth 6 -type d | grep -v release # expect no output

- Inspected a few snippets from various places - they look OK.
Re-checked-in automatically generated patcher config, kept 'beta-dir beta' line.

Merge snippets

Merged snippets + partials into one directory with the following commands:

mkdir 3.0rc3-3.0.1
rsync -av 3.0rc3-3.0.1.rc3.mac/ 3.0rc3-3.0.1/
rsync -av 3.0rc2-3.0.1/ 3.0rc3-3.0.1/

Restarted automation at Updates::Push

Automation continued normally.

Publish Updates to Test Channels (betatest & releasetest)

No problems

Update Verify

Linux passed.
Mac had expected failure of gu-IN (because we stopped shipping it on Mac at rc1)
Win32 had expected failures about binary files

Stage

No problems

Push Updates to Beta Channels

-bash-3.2$ time ~/bin/backupsnip 20080703-Firefox-3.0.1-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080703-4-pre-20080703-Firefox-3.0.1-beta.tar.bz2 .

real	28m44.170s
user	0m22.740s
sys	0m33.067s
-bash-3.2$ time ~/bin/pushsnip 20080703-Firefox-3.0.1-beta

sent 737264 bytes  received 44534 bytes  10354.94 bytes/sec
total size is 588339  speedup is 0.75

real	1m15.405s
user	0m0.107s
sys	0m3.612s

Sign Installers

Done manually using these installer-signing-instructions here

complete stage-merged:

# on stage
cd /data/cltbld/firefox-3.0.1/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/

Create MD5 and SHA1 checksum files

# on stage
cd /data/cltbld/firefox-3.0.1/stage-merged/
~/bin/checksum-files .

Fix permissions & ownership (on the two SUM files, and the detached sigs)

chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

Manually added entries to download.mozilla.org/admin.php
Linux/win32 partials were added under the 'Firefox-3.0.1-Partial-3.0rc2' product with filenames like:

firefox-3.0rc3-3.0.1.partial.mar

Mac partials use 'Firefox-3.0.1-Partial-3.0rc3'
Had to do this because we only shipped rc3 on Mac. Stage step renamed partials to 3.0rc3-3.0.1 but snippets point to, eg. http://download.mozilla.org/?product=firefox-3.0.1-partial-3.0rc2&os=win&lang=af

(Thanks Nick)

Push to mirrors

push the stage-merged directory to the releases area:

# on stage
rsync -av /data/cltbld/firefox-3.0.1/stage-merged/ /home/ftp/pub/firefox/releases/3.0.1/

Final Verification

Verify that releasetest points to valid bouncer links:

# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log

Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):

grep HTTP quickVerify.log | grep -v 200 | grep -v 302

Two 404s caused by mirrors in a half-synced state
One 500 error, seems transient

Before pushing final updates,verify that "release" and "releasetest" channel match:

# on aus2-staging
cd /opt/aus2/snippets/staging/20080703-Firefox-3.0.1
find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080703-Firefox-3.0.1-test/$a");'

Publish Updates to Release Channel

-bash-3.2$ time ~/bin/backupsnip 20080703-Firefox-3.0.1
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080716-2-pre-20080703-Firefox-3.0.1.tar.bz2 .
real	30m39.986s
user	0m23.497s
sys	0m34.947s
-bash-3.2$ time ~/bin/pushsnip 20080703-Firefox-3.0.1

Release

# on stage as cltbld
cd /home/ftp/pub/firefox/releases
rm latest
ln -s latest-3.0 latest
rm latest-3.0
ln -s 3.0.1 latest-3.0

Module	Branch	Tag	Pull date
cvsroot/mozilla	HEAD	FIREFOX_3_0_1_BUILD1	2008-07-02 01:52 PDT
l10n/l10n	HEAD	FIREFOX_3_0_1_BUILD1	2008-07-02 01:52 PDT

Releases/Firefox 3.0.1:BuildNotes

Contents

Build Engineers

Bonsai queries

Tags

Build data

Notes

Build 1

Tag

Source

Build

Repack

Sign

L10nVerify

Generate Updates

Mac snippets (3.0rc3 -> 3.0.1)

Linux & win32 snippets (3.0rc2 -> 3.0.1)

Merge snippets

Publish Updates to Test Channels (betatest & releasetest)

Update Verify

Stage

Push Updates to Beta Channels

Sign Installers

Update Bouncer

Push to mirrors

Final Verification

Publish Updates to Release Channel

Release

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools