Releases/Firefox 3.0.2:BuildNotes

From MozillaWiki
Jump to: navigation, search

Build Engineers

joduinn
Version/config bump bug

Bonsai queries

Updated CVS Tags devmo page.

Tags

Build 1:

Module Branch Tag Pull date
cvsroot/mozilla HEAD FIREFOX_3_0_2_BUILD1 2008-08-27 01:31 PDT
l10n/l10n HEAD FIREFOX_3_0_2_BUILD1 2008-08-27 01:31 PDT

Build 2:

Module Branch Tag Pull date
cvsroot/mozilla GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD2 N/A
l10n/l10n GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD2 N/A

Build 3:

Module Branch Tag Pull date
cvsroot/mozilla GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD3 N/A
l10n/l10n GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD3 N/A

Build 4:

Module Branch Tag Pull date
cvsroot/mozilla GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD4 N/A
l10n/l10n GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD4 N/A

Build 5:

Module Branch Tag Pull date
cvsroot/mozilla GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD5 N/A
l10n/l10n GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD5 N/A

Build 6:

Module Branch Tag Pull date
cvsroot/mozilla GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD6 N/A
l10n/l10n GECKO190_20080827_RELBRANCH FIREFOX_3_0_2_BUILD6 N/A

Build data

Type Build ID SHA1 Push date Build machine
[Windows installer/zip] fx-win32-1.9-slave2
[Mac compressed] fx-mac-1.9-slave2
[Linux compressed] fx-linux-1.9-slave2

Notes

Last minute discovery of bug 380994comment#25, and bug 360529comment#60 were both discovered *before* builds started, but delayed "formal go to build".

Build 1

Bootstrap Tag: RELEASE_AUTOMATION_M11

Setup before starting:

  • Updated master.cfg, bootstrap.cfg. Retagged bootstrap.cfg.
  • On the slaves:
    • ran 'DISPLAY=:0 xhost +' to make sure linux AliveTest works.
    • remove /builds/verify/firefox-3.0.1
    • update tinderbox to RELEASE_AUTOMATION_M10
      • For win32:
cd /e/builds/tinderbox/mozilla/tools/tinderbox
cvs up -r RELEASE_AUTOMATION_M11
for dir in fx19rel fx19l10nrel xr19rel; do 
  cd /e/$dir; 
  for f in `ls *.pl *.html INSTALL README | grep -v tinder-config.pl`; do 
    cp -pv /e/builds/tinderbox/mozilla/tools/tinderbox/$f .;
  done
  cp -pv post-mozilla-rel.pl post-mozilla.pl
done
    • On the linux slave, removed:
      • /builds/updates/firefox-3.0.1 and /builds/updates/3.0.2-test
      • /builds/source/*
      • /data/cltbld/thunderbird-3.0a2*
    • On the mac slave, removed:
      • /builds/tinderbox/Tb-Mozilla1.9-Release/Darwin_8.8.4_Depend
      • /builds/tinderbox/Tb-Mozilla1.9-l10n-Release/Darwin_8.8.4_Depend
      • /builds/tinderbox/Xr-Mozilla1.9-Release/Darwin_8.8.4_Depend
  • Space on slaves before starting:
    • fx-linux-1.9-slave2: 24G on /builds
    • fx-mac-1.9-slave2: 24G on /
    • fx-win32-1.9-slave2: 17G on d:, 5G on e: (disk heavy work is done on d, eg update_verify. build/repack is done on e, but mostly just overwrites existing data)
  • Gulped, and kicked off automation:
buildbot sendchange --username=bhearsum --master=localhost:9989 -m"Firefox 3.0.2build1 release" gogogo

Tag

Automated, no problems.

Source

Automated, no problems.

respin declared for bug 452385, follow up regression fix to bug 433340

Build 2

Code errors were found after we tagged for build 1, therefore we had to spin a build 2.

Bootstrap Tag: RELEASE_AUTOMATION_M11

  • Landed bootstrap bumps for the respin

Tag

Automated, no problems.

Source

Automated, no problems

Build

Automated, no problems.

Repack

No problems on linux and mac. Win32 failed out in Repack::Execute because the E: drive was full. It had finished the repacks, but failed to push to stage. Cleared space (deleted the Tb and Xr release build directories) and restarted from Repack::Execute.

Sign

  • Signing bailed out during 'ro' with this error:
*** Resign list hit; resigning /home/cltsign/signing-work/firefox-3.0.2/tmp/RepackExe-JeG9pR/localized/uninstall/helper.exe...
Signing /home/cltsign/signing-work/firefox-3.0.2/tmp/RepackExe-JeG9pR/localized/uninstall/helper.exe...
Error: Invalid timestamp http address
Error: TimeStamping Failed.  Result = 80070001, (-2147024895)
Failed to sign /home/cltsign/signing-work/firefox-3.0.2/tmp/RepackExe-JeG9pR/localized/uninstall/helper.exe; bailing

respin declared for bug 441087

Build 3

Tag

No Problems

Source

No problems

Build

No problems

Repack

No problems

Sign

no problems

L10nVerify

L10nverify spewed tons of output from updater.ini differences. It turns out that https://bugzilla.mozilla.org/show_bug.cgi?id=446527 busted updater.ini on linux and mac. There was win32 about as well, which was completely legitimate (because Locale= was newly added to updater.ini for win32).

respin declared for bug 446527

Build 4

Tag

  • Automated, no problems.

Source

  • Automated, no problems.

Build

  • Automated, no problems.

Repack

  • Automated, no problems.

Sign

instructions
Hit timestamp problem again. To work-around, did the following (from https://bugzilla.mozilla.org/show_bug.cgi?id=415966#c25): 

mkdir signed-build4-partial unsigned-build4-partial
cp signed-build4/*en-US* signed-build4-partial/
cp unsigned-build4/*en-US* unsigned-build4-partial/
# copied everything from it -> zh-TW from unsigned-build4 to unsigned-build4-partial
perl ./signing/sign-release.pl -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} -p 2>&1 | tee -a win32_signing_build${BUILD}.log
# to merge the signed builds back together
rsync -av signed-build4-partial/ signed-build4/
# then push the builds back to stage (instructions on the signing doc)

L10nVerify

Some string changes in updater.ini - this is legitimate.

Generate Updates

  • updates::execute has been run, automation needs to be restarted from updates::push (after 2.0.0.17 test snippets are done being pushed)
  • locally modified master.cfg to skip over updates::execute and start at updates::push. Now in progress.

Publish Updates to Test Channels (betatest & releasetest)

  • manually forced step from waterfall. No problems, worked fine.

Update Verify

Stage

  • manually forced step from waterfall. No problems, worked fine.

respin declared because of bug 453227

Build5

Manual cleanup skipped. Had fun restarting master because master.cfg was accidently refreshed from cvs earlier this morning, losing all the passwords. Logged into production-1.9-master, as buildbot, and ran the following: 

buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.2build5 release" gogogo

Tag

  • Automated, no problems.

Source

  • Automated, no problems.

Build

  • Automated, no problems.

Repack

  • Automated, no problems.

Sign

instructions

L10nVerify

  • Automated, no problems. Expected fails for gu-IN during the FF3.0betas

Generate Updates

  • Automated, no problems.

Publish Updates to Test Channels (betatest & releasetest)

  • Automated, no problems.

Update Verify

  • Automated, no problems.

Stage

  • Automated, no problems.

Push to beta channel

While waiting for QA to give "ok", on aus2-staging do: 

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
$  time ~/bin/backupsnip 20080908-Firefox-3.0.2-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080910-1-pre-20080908-Firefox-3.0.2-beta.tar.bz2 .
real    30m49.454s
user    0m34.069s
sys     0m37.876s
$
$ time ~/bin/pushsnip 20080908-Firefox-3.0.2-beta
sent 858677 bytes  received 49674 bytes  21372.96 bytes/sec
total size is 687093  speedup is 0.76
real    0m42.322s
user    0m0.051s
sys     0m4.122s

Respin declared for bug 454406, bug 443918 and bug 455283

Build6

Manual cleanup skipped. Logged into production-1.9-master, as buildbot, and ran the following: 

buildbot sendchange --username=joduinn --master=localhost:9989 -m"Firefox 3.0.2build6 release" pleasenomorerespins

Tag

  • Automated, no problems.

Source

  • Automated, no problems.

Build

  • Automated, no problems.

Repack

  • Automated, no problems.

Sign

instructions

Signing failed out on 'de' with: 

 Error: Invalid timestamp http address
 Error: TimeStampign Failed. Result = 80070001, (-2147024895)

To work-around, did the following (from https://bugzilla.mozilla.org/show_bug.cgi?id=415966#c25): 

mkdir signed-build6-partial unsigned-build6-partial
cp signed-build6/*en-US* signed-build6-partial/
cp unsigned-build6/*en-US* unsigned-build6-partial/
# copied everything from 'de' -> 'zh-TW' from unsigned-build6 to unsigned-build6-partial
perl ./signing/sign-release.pl -o signed-build${BUILD}-partial -M dir -d unsigned-build${BUILD}-partial -a ${PRODUCT} --release ${TAG} 2>&1 | tee -a win32_signing_build${BUILD}.log
# to merge the signed builds back together
rsync -av signed-build6-partial/ signed-build6/
# then push the builds back to stage (instructions on the signing doc)

L10nVerify

Generate Updates

Need to bring people who already updated to 3.0.2 build5 up to build6, so do this (after the automated run) 

# connect as cltbld@fx-linux-1.9-slave2
cd /builds/updates/firefox-3.0.2/patcher/temp/firefox/3.0.1-3.0.2
for dir in aus2.test aus2.beta aus2; do
  # copy complete snippets from 3.0.1 into a new dir for 3.0.2
  mkdir -p b5-b6-${dir}/Firefox
  rsync -a --exclude 'partial.txt' ${dir}/Firefox/3.0.1/ b5-b6-${dir}/Firefox/3.0.2/
  # use the buildid's from build5
  mv b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008070206  b5-b6-${dir}/Firefox/3.0.2/Darwin_Universal-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008070206         b5-b6-${dir}/Firefox/3.0.2/Linux_x86-gcc3/2008090512
  mv b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008070208         b5-b6-${dir}/Firefox/3.0.2/WINNT_x86-msvc/2008090514
done
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-test
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6-beta
rsync -a -e 'ssh -i ~/.ssh/aus' b5-b6-aus2/      cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080919-Firefox-3.0.2build5-build6

Then go to aus2-staging and push 20080919-Firefox-3.0.2build5-build6-test live.

Publish Updates to Test Channels (betatest & releasetest)

  • Automated, no problems.

Update Verify

  • linux no problems
  • win32 expected problems (known checksum dif bug)
  • mac expected problems (added/dropped locales)

Stage

  • Automated, no problems.

Push to beta channel

Backup on aus2-staging doing: 

 # put snippets on beta
 $ sudo su - cltbld
 # make sure using latest version of scripts in mozilla/tools/release/bin/
 $ cd bin
 $ cvs update . 
 $ cd /opt/aus2/snippets/staging
 # note the required parameter must match what will be used with pushsnip below. 
$  time ~/bin/backupsnip 20080917-Firefox-3.0.2-beta
Running /bin/tar cfvj /opt/aus2/snippets/backup/20080910-1-pre-20080917-Firefox-3.0.2-beta.tar.bz2 .
real    31m55.896s
user    0m37.535s
sys     0m50.817s
$
$ time ~/bin/pushsnip 20080917-Firefox-3.0.2-beta
real    1m17.380s
user    0m0.097s
sys     0m4.970s
$ time ~/bin/pushsnip 20080919-Firefox-3.0.2build5-build6-beta
real    0m8.604s
user    0m0.009s
sys     0m0.441s

NB: Had to push both 20080917-Firefox-3.0.2-beta and 20080919-Firefox-3.0.2build5-build6-beta directories.

Sign Installers

Done manually using these installer-signing-instructions here

  • complete stage-merged:
# on stage
cd /data/cltbld/firefox-3.0.2/
rsync -av batch1/mar/ stage-merged/
rsync -av batch1/stage-signed/ stage-merged/
  • Create MD5 and SHA1 checksum files
# on stage
cd /data/cltbld/firefox-3.0.2/stage-merged/
~/bin/checksum-files .
  • Fix permissions & ownership (on the two SUM files, and the detached sigs)
chown -R cltbld:firefox .
chmod 644 *SUMS

Update Bouncer

  • Manually added entries to download.mozilla.org

Push to mirrors

  • push the stage-merged directory to the releases area:
# on stage
rsync -av /data/cltbld/firefox-3.0.2/stage-merged/ /home/ftp/pub/firefox/releases/3.0.2/

Final Verification

  • Verify that releasetest points to valid bouncer links:
# this can be run from anywhere
cvs co mozilla/testing/release
cd mozilla/testing/release/updates
cat moz19-firefox-*.cfg | grep -v major | sed 's/betatest/releasetest/' | grep -v 2.0a | grep -v 2.0b > update.cfg
./verify.sh -t update.cfg 2>&1 | tee quickVerify.log
  • Look for any HTTP error codes besides 200 ("OK") and 302 ("Found"):
grep HTTP quickVerify.log | grep -v 200 | grep -v 302
    • 23 occurrences of 404s
  • Before pushing final updates,verify that "release" and "releasetest" channel match:
# on aus2-staging
$ cd 20080917-Firefox-3.0.2
$ find -type d -iregex '.*release.*' | perl -nle '$a = $_; $a =~ s/release/releasetest/; system("diff -r -u $_ ../20080917-Firefox-3.0.2-test/$a");'
$

Publish Updates to Release Channel

While waiting for formal "go", do: 

$ time ~/bin/backupsnip 20080917-Firefox-3.0.2
real    33m24.621s
user    0m38.584s
sys     0m41.689s

$ time ~/bin/pushsnip 20080917-Firefox-3.0.2
real    0m17.352s
user    0m0.027s
sys     0m1.464s
$ time ~/bin/pushsnip 20080919-Firefox-3.0.2build5-build6
real    0m8.355s
user    0m0.004s
sys     0m0.329s

NB: Be sure to push both 20080917-Firefox-3.0.2 and 20080919-Firefox-3.0.2build5-build6 directories.

Release

  • On stage.m.o, do the following:
    • edit the exclude file /pub/mozilla.org/zz/rsyncd-mozilla-current.exclude to add the new release.
    • Update "latest" symlinks 
$ cd /pub/mozilla.org/firefox/releases/
$ mv latest-3.0 latest-3.0.old
$ ln -s 3.0.2 latest-3.0
$ rm latest-3.0.old
Retrieved from "https://wiki.mozilla.org/index.php?title=Releases/Firefox_3.0.2:BuildNotes&oldid=147459"