Releases/Firefox 3.5.17/BuildNotes

From MozillaWiki
Jump to: navigation, search

Notes About Releasing

Please update the Release:Primer for future releases (bug fixes, changes to automation) as needed

Build Engineers

bhearsum/coop
bug 624494

Bugs hit

  • bug 626164 - Clobber won't work for build1
  • bug 637886 - throttling patch not ready ahead of time
  • bug 629256 - we disabled MU for broken locales for build 1, but then didn't do the same for build 2 and build 3. Needed to push out the unmodified MU list due to time constraints, so broken locales will continue to be broken.

Signed-off Revision(s)

Build 1: bc91b067f42f

L10N changesets

Updates l10n-changesets obtained from the l10n shipping dashboard.

Tags

Build # Tag Changeset
1 FIREFOX_3_5_17_RELEASE FIREFOX_3_5_17_BUILD1 bc91b067f42f

Build data

Build # Type Build ID Build machine Time to build
1 Linux 20110121161846 linux-ix-slave14 29 mins, 15 secs
Mac 20110121150727 bm-xserve11 1 hrs, 41 mins, 34 secs
Windows 20110121150729 mw32-ix-slave02 1 hrs, 18 mins, 25 secs

Notes

Done on pm01

Build 1

  • Set reserved slaves to 4:
# cltbld@pm01
echo 4 > /builds/buildbot/builder_master1/reserved_slaves_pm01
  • Clobber won't work for build1 because of bug 626164
    • Looked around all of the fast slaves and didn't see anything lying around, so they were clobbered at some point already.
  • Landed updated automation config
  • Manually tagged buildbot-configs, buildbotcustom, tools with FIREFOX_3_5_17_RELEASE_FIREFOX_3_5_17_BUILD1
  • Updated buildbot-configs/buildbotcustom
  • Ran release_sanity.py w/ the dry run option:
PYTHONPATH=.:tools/lib/python/ python release_sanity.py -u bhearsum -V 3.5.17 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py --dryrun localhost:9010
  • After it passed, ran it without --dry-run to start the release process:
PYTHONPATH=.:tools/lib/python/ python release_sanity.py -u bhearsum -V 3.5.17 --branch mozilla-1.9.1 --build-number 1 -c release-firefox-mozilla-1.9.1.py localhost:9010

Tag

No problems

Source

No problems

Build

  • Linux failed to upload symbols
    • Clobbered (manually) and re-kicked with "Force Build"
  • Linux failed to upload symbols again
    • Catlee pointed out that the host key verification was failing
    • Logged onto the slave, manually clobbered, fixed the hostkey with:
# cltbld@linux-ix-slave14
ssh -oBatchMode=no dm-symbolpush01.mozilla.org
# Typed "yes" when prompted
    • Started a new build with "Force Build"
  • No problems with Mac or Windows

L10n Repack

  • No problems on Mac or Windows
  • Had to use "Force Build" with the following properties to start the Linux ones, because of the Linux en-US issues:
release_config: mozilla/release-firefox-mozilla-1.9.1.py
script_repo_revision: FIREFOX_3_5_17_RELEASE

XULRunner Source & Build

No problems

Sign

Removed en-US.xpi from candidates directory

Ran the following on the signing machine

PRODUCT=firefox
VERSION=3.5.17
BUILD=1
TAG=FIREFOX_3_5_17_RELEASE
REPO='releases/mozilla-1.9.1'
EMAIL=bhearsum@mozilla.com
KEYDIR=d:/2010-keys
cd ~/hg-tools
hg pull && hg up -r ${TAG}
mkdir -p ~/signing-work/${PRODUCT}-${VERSION}/lib/python
cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .
rsync -av ~/hg-tools/lib/python/ ./lib/python/
make autosign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

And then to upload:

make postsign PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL}

Signature verification succeeded.

L10n Verify

  • Differences for all locales/platforms because of the shortened build directories.

Updates

  • First attempt failed with an ssh key error, on the same slave that had key issues during the Linux build.
    • Replaced keys on it with ones from a known good slave:
# cltbld@linux-ix-slave14
su -c 'umount /home/cltbld/.ssh' # multiple times, until fully unmounted
rm -rf .ssh
scp -r cltbld@mv-moz2-linux-ix-slave08:.ssh .
  • Restarted updates with "Force Build"
    • No problems
  • the updates builder was triggered a third time, but not sure by what/who. It failed early, and should be harmless.

Update Verify

Triggered by hand.

  • Linux:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5\
b99/20090605144815/Linux_x86-gcc3/ga-IE/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.5b99/20090605144815/Linux_x8\
6-gcc3/ga-IE/betatest/update.xml?force=1
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.\
5b99/20090605144815/Linux_x86-gcc3/ga-IE/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
  • Mac:
FAIL: no partial update found for https://aus2.mozilla.org/update/1/Firefox/3.5\
.1/20090715083437/Darwin_Universal-gcc3/de/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1
Using  https://aus2.mozilla.org/update/1/Firefox/3.5.1/20090715083437/Darwin_Un\
iversal-gcc3/de/betatest/update.xml?force=1
2011-01-24 15:55:38 URL:http://stage-old.mozilla.org/pub/mozilla.org/firefox/ni\
ghtly/3.5.17-candidates/build1/update/mac/de/firefox-3.5.17.complete.mar [17980\
405/17980405] -> "update/complete.mar" [1]
FAIL: partial updates are larger than complete updates
  • Win32:
FAIL: no complete update found for https://aus2.mozilla.org/update/1/Firefox/3.\
5.10/20100504093643/WINNT_x86-msvc/pt-BR/betatest/update.xml?force=1
FAIL: download_mars returned non-zero exit code: 1

Socorro product data updates

Added '3.5.17' and '3.5.18pre' to crash-stats with this admin page.

Update Bouncer

Added by forcing bouncer_submitter builder

Major Update Generation (-> 3.6.14)

Used 'force build' to start the major_update builder.

Had more problems with linux-ix-slave14. It turns out that this slave was still syncing with staging-puppet, which is why we were having ssh key issues.

I fixed up the slave to sync with production-puppet and the problem went away.

Major Update Verification

  • mn fails for all platforms, normal and expected.
  • sl blurts the following, normal and expected:
Only in source/firefox/searchplugins: iskalnik-gov-si.xml
Only in source/firefox/searchplugins: najdi-si.xml
Only in source/firefox/searchplugins: yahoo.xml
  • Linux:
  • some path differences:
    • ru:
Only in source/firefox/searchplugins: rambler.xml
  • Mac: the usual mac differences:
diff -r source/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js target/Firefox.app/Contents/MacOS/defaults/pref/channel-prefs.js
1c1
 < //@line 2 "/builds/moz2_slave/release-mozilla-1.9.1-macosx_build/build/browser/app/profile/channel-prefs.js"
---
 > //@line 2 "/builds/slave/release-mozilla-1.9.2-macosx_build/build/browser/app/profile/channel-prefs.js"
Only in source/Firefox.app/Contents: Plug-Ins
 source/Firefox.app/Contents/Plug-Ins/PrintPDE.plugin/Contents/Resources/English.lproj/PrintPDE.nib
  • Windows: the usual .chk file differences:
Only in source/bin: .autoreg
diff -r source/bin/defaults/pref/channel-prefs.js target/bin/defaults/pref/channel-prefs.js
1c1
< //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.1-win32_build\build\browser\app\profile\channel-prefs.js"
---
> //@line 2 "e:\builds\moz2_slave\release-mozilla-1.9.2-win32_build\build\browser\app\profile\channel-prefs.js"
Files source/bin/freebl3.chk and target/bin/freebl3.chk differ
Files source/bin/nssdbm3.chk and target/bin/nssdbm3.chk differ
Only in source/bin: removed-files
Files source/bin/softokn3.chk and target/bin/softokn3.chk differ

Publish updates to beta channel

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20110121-Firefox-3.5.17-beta
 real    35m9.724s
 user    0m2.357s
 sys     0m23.525s
time ~/bin/pushsnip 20110121-Firefox-3.5.17-beta
 real    29m36.768s
 user    0m2.182s
 sys     0m34.530s

Final checks before push

# ffxbld@stage
cd /pub/mozilla.org/firefox/nightly/3.5.17-candidates/build1/
find . ! -user ffxbld # (anything in contrib/ is ok)
find . ! -group firefox
find . -type f ! -perm 644
find . -mindepth 1 -type d ! -perm 755 ! -name 'contrib*'
find . -maxdepth 1 -type d ! -perm 2775  -name 'contrib*'

contrib/solaris_pkgadd and solaris_tarball were 755 (g+s), which is OK too.

Virus check

# cltbld@stage
clamdscan /mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.17-candidates/build1/
/mnt/netapp/stage/archive.mozilla.org/pub/firefox/nightly/3.5.17-candidates/build1: OK

----------- SCAN SUMMARY -----------
Infected files: 0
Time: 2694.758 sec (44 m 54 s)

Major Update Generation (-> 3.6.14 build2)

Used 'force build' to start the major_update builder.

No problems

Major Update Verification

Same "failures" as for build1.

Major Update Generation (-> 3.6.14 build3)

Used 'force build' to start the major_update builder.

No problems

Major Update Verification

Same "failures" as for build1.

Pre-push Checks

Triggered manually from waterfall.

  • Revision to build: FIREFOX_3_5_17_RELEASE
  • Property 1 name -> release_config: mozilla/release-firefox-mozilla-1.9.1.py

Push to mirrors

Used the new automatic builder. Failed a few times before realizing I needed to reconfig the master and re-tag the tools repo to pick up the changes (done by Rail)

No problems after that.

Final verify

A few 404s, but nothing serious.

Push updates to release channel

Please use the

  • 20110126-Firefox-3.5.17-bug629256fix directory when pushing to the release channel for 3.5.x -> 3.5.17 updates
  • 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix directory when pushing the major update from 3.5.17 to 3.6.14

for bug 629256. We drop updates for a few locales which have strings that make the updater crash. NThomas 23:52, 26 January 2011 (PST)

# cltbld @ aus2-staging
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20110126-Firefox-3.5.17-bug629256fix

time ~/bin/pushsnip 20110126-Firefox-3.5.17-bug629256fix
 real    17m24.216s
 user    0m6.373s
 sys     0m34.334s

Push 3.5.17 -> 3.6.14 MU

Forgot to roll a version update patch for throttling. Got that landed and then pushed in bug 637886.

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix
~/bin/pushsnip 20110126-Firefox-3.5.17-3.6.14-MU-bug629256fix

But those were the 3.6.14build1 snippets, fix:

# cltbld@aus2-staging
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20110222-Firefox-3.5.17-3.6.14-MU
~/bin/pushsnip 20110222-Firefox-3.5.17-3.6.14-MU

Release

# ffxbld@stage
cd /home/ftp/pub/firefox/releases/
rm latest-3.5 && ln -s 3.5.17 latest-3.5

Sent the following to metrics@mozilla.com

Firefox,firefox,firefox,3.5.17,3.5
Firefox,firefox,firefox,3.5.18pre,3.5

FIREFOX DONE TO HERE

XULRunner

Builds

  • Linux: No problems.
  • Windows: No problems.
  • Mac: No problems.

Signing

Signed with:

PRODUCT=xulrunner
VERSION=1.9.1.17
BUILD=1
TAG=FIREFOX_3_5_17_RELEASE
REPO="releases/mozilla-1.9.1"
EMAIL=coop@mozilla.com
KEYDIR=d:/2010-keys

mkdir -p ~/signing-work/${PRODUCT}-${VERSION}
cd ~/signing-work/${PRODUCT}-${VERSION}
cd ~/hg-tools
hg pull
hg up -r ${TAG}

cd ~/signing-work/${PRODUCT}-${VERSION}
cp ~/hg-tools/release/signing/* .

make setup PRODUCT=${PRODUCT} VERSION=${VERSION} \
  BUILD=${BUILD} REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1
rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  --exclude=*.txt --exclude=*-symbols.zip \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/ \
 unsigned-build${BUILD}
rsync -av --exclude=unsigned unsigned-build${BUILD}/ \
 signed-build${BUILD}/
rsync -av unsigned-build${BUILD}/unsigned/ \
 signed-build${BUILD}/

make checksum-files create-sigs stage verify-sigs \
  PRODUCT=${PRODUCT} VERSION=${VERSION} BUILD=${BUILD} \
  REPO=${REPO} EMAIL=${EMAIL} KEYDIR=${KEYDIR} USE_NEW=1

rmdir signed-build${BUILD}/contrib{,-localized}

rsync -av -e "ssh -i /home/cltsign/.ssh/xrbld_dsa" \
  signed-build${BUILD}/ \
  xrbld@stage.mozilla.org:/home/ftp/pub/${PRODUCT}/nightly/${VERSION}-candidates/build${BUILD}/

Push XULRunner to mirrors

  1. xrbld@stage
rsync -av --exclude=*.txt --exclude=*unsigned* --exclude=*symbols.zip \
/pub/mozilla.org/xulrunner/nightly/1.9.1.17-candidates/build1/  \
/pub/mozilla.org/xulrunner/releases/1.9.1.17/

Edited cltbld@stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude and added

- xulrunner/releases/1.9.1.16

...to exclude the previous release.

Update XULRunner wiki

Update the links in these two wiki pages for the version change. NOTE: make sure to update all the links on the page!!!

XR DONE TO HERE

MU to 3.6.17

rail forced the MU last night.

We had to do a fast 3.6.15 release so we had to generate a new MU

# cltbld@aus2
# From any build slave
cd /opt/aus2/snippets/staging
~/bin/backupsnip 20110303-Firefox-3.5.17-3.6.15-MU-beta
~/bin/backupsnip 20110303-Firefox-3.5.17-3.6.15-MU
~/bin/pushsnip 20110303-Firefox-3.5.17-3.6.15-MU-beta
~/bin/pushsnip 20110303-Firefox-3.5.17-3.6.15-MU