Release Engineers

Nick Thomas (major update test)
Ben Hearsum (real snippet generation, null partner updates)

Bugs

bug 442105 - tracking bug
bug 450863 - null partner updates

Generate Major Update Snippets

# ssh cltbld@prometheus-vm.build.mozilla.org
cd /builds/2.0.0.16-major-update/snippets
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d patcher mozilla/tools/patcher
cd patcher
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d MozBuild mozilla/tools/release/MozBuild
cd ../
cvs -d:ext:ffxbld@cvs.mozilla.org:/cvsroot co -d patcher-configs mozilla/tools/patcher-configs

cd patcher
export CVSROOT=:ext:ffxbld@cvs.mozilla.org:/cvsroot

# build tools
./patcher2.pl --build-tools --app=firefox --tools-rev=UPDATE_PACKAGING_R4 --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_build-tools.log

# download mars
./patcher2.pl --download --app=firefox --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox_download.log

# FIXME - patcher needs to see that the MARs that it thinks are partials
#             are there or else it will not attempt to generate patchinfo
cd temp/firefox
ln -s 3.0.1 2.0.0.16-3.0.1

# Create partial patches and snippets
./patcher2.pl --create-patches --app=firefox --config=../patcher-configs/moz18-branch-major-update-patcher2.cfg 2>&1 | tee ../firefox-create-patches.log

Generate Null Partner Snippets

Create Partner Channel List

Starting with https://bugzilla.mozilla.org/attachment.cgi?id=334332

# Removed any channels that don't start with 'release-' from the list
# Now, we only have a list of the real release channels for partner builds
# we need to add beta and test channels to the list:
for i in `cat partner-update-channels`; do basechan=`echo $i | sed -e 's/release-//'` && echo "beta-$basechan" >> partner-update-channels && echo "betatest-$basechan" >> partner-update-channels && echo "releasetest-$basechan" >> partner-update-channels; done
# now, we have all the channels we need except for the 4 listed at the start
# because the real release channels aren't prepended with 'release' we will do this different:
for i in [list of channels that don't start with release-]; do echo $i >> partner-update-channels && echo "beta-$i" >> partner-update-channels && echo "betatest-$i" >> partner-update-channels && echo "releasetest-$i" >> partner-update-channels; done
# now we have a complete list of partner channels to generate snippets for

Generate null snippets

updated null_partners.sh with 2.0.0.16 buildids, and enabled it to read a list of partner channels from an external file - rev 1.3

# ssh cltbld@prometheus-vm.build.mozilla.org
cd /builds/2.0.0.16-major-update/snippets/patcher/temp/firefox/2.0.0.16-3.0.1
bash ../../../one-offs/null_partners.sh ../../../partner-update-channels

Create beta snippet dir

rsync -a aus2/ aus2.beta/

Trim snippets from release, beta, test dirs

Our 'aus2.test' directory should only contain 'releasetest-*' and 'betatest-*' snippets. Similarly, 'aus2.beta' should only contain 'beta-*' snippets and 'aus2' should only have 'release-*'

# beta
find aus2.beta/ -type d -name "release*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "releasetest*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "betatest*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
# these ones get rid of those funny channels
find aus2.beta/ -type d -name "default*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "nightly*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
find aus2.beta/ -type d -name "Windows*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log
# release 
find aus2/ -type d -name "beta*" -exec rm -rfv {} \; 2>&1 | tee -a release_remove.log
find aus2/ -type d -name "releasetest*" -exec rm -rfv {} \; 2>&1 | tee -a release_remove.log
find aus2/ -type d -name "betatest*" -exce rm -rfv {} \; 2>&1 | tee -a release_remove.log
# test
find aus2.test/ -type d -name "release-*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
find aus2.test/ -type d -name "beta-*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
# these ones get rid of those funny channels
find aus2.test/ -type d -name "default*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
find aus2.test/ -type d -name "nightly*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log
find aus2.test/ -type d -name "Windows*" -exec rm -rfv {} \; 2>&1 | tee -a test_remove.log

Move null snippets to their own directory

mkdir aus2.null
rsync -av aus2/ aus2.null/
rsync -av aus2.beta/ aus2.null/
rsync -av aus2.test/ aus2.null/
# trim non-partner snippets from the null directory
find aus2.null -type d -mindepth 6 -maxdepth 6 | egrep '.*\/(release|releasetest|beta|betatest)$' | xargs rm -rfv 2>&1 | tee -a ../null_remove.log
# trim partner snippets from the other directories
find aus2.test -type d -mindepth 6 -maxdepth 6 | egrep '.*\/(release|beta)test-.*$' | xargs rm -rfv 2>&1 | tee -a ../test_remove.log
find aus2.beta -type d -mindepth 6 -maxdepth 6 | egrep '.*\/beta-.*$' | xargs rm -rfv 2>&1 | tee -a ../beta_remove.log
find aus -type d -mindepth 6 -maxdepth 6 | egrep -v '.*\/release$' | xargs rm -rfv 2>&1 | tee -a ../release_remove.log 

Verify correct snippets are in correct directories

Make sure that only betatest/releasetest snippets are in aus2.test, beta snippets in aus2.beta, and release snippets in aus2
# aus2.null
find aus2.null -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep '^(release-*|releasetest-*|beta-*|betatest-*)'
# aus2.test
find aus2.test -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep -v '^(beta|release)test$'
# no output
# beta
find aus2.beta -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep -v '^beta$'
# no output
# release
find aus2 -type d -mindepth 6 -maxdepth 6 | cut -d / -f 7 | egrep -v '^release$'

Push snippet directories to aus2-staging

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2 cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2-beta cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU-beta
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2-test cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU-test
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2-null cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080820-Firefox-2.0.0.16-3.0.1-MU-null

Push null & test snippets live

cd /opt/aus2/snippets/staging
~/bin/backupsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-null
~/bin/pushsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-null
~/bin/backupsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-test
~/bin/pushsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-test

Fix up null partner snippets to target 2.0.0.14

After partner snippets were pushed I was informed that the latest partner builds we shipped were for 2.0.0.14. Therefore, the snippets pushed are useless. To fix them up, did the following:

# cltbld@prometheus-vm
cd /builds/2.0.0.16-major-update/snippets/patcher/temp/firefox/2.0.0.16-3.0.1
cd aus2.null/Firefox
mv 2.0.0.16 2.0.0.14
cd 2.0.0.14/Darwin_Universal-gcc3
mv 2008070205 2008040413
cd ../Linux_x86-gcc3
mv 2008070204 2008040414
cd ../WINNT_x86-msvc
mv 2008070205 2008040413

Also had to generate snippets for another partner channel for 2.0.0.16 (https://bugzilla.mozilla.org/show_bug.cgi?id=450863#c5) To do that, did the following:

cd /builds/2.0.0.16-major-update/snippets/patcher
echo "release-cck-asus-google" >> additional-partner-channels
echo "releasetest-cck-asus-google" >> additional-partner-channels
echo "beta-cck-asus-google" >> additional-partner-channels
echo "betatest-cck-asus-google" >> aditional-partner-channels
cd temp/firefox/2.0.0.16-3.0.1
# modified null_partners.sh to only generage snippets in aus2.null
bash ../../../one-offs/null_partners.sh ../../../additional-partner-channels

Did a spot check of aus2.null, looked good. Pushed new snippets to aus2-staging:

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.null cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-null-partner-snippets

Verify null partner updates

Modified create_aus_urls.txt to use a different set of locales for each platform and read partner channels from a file. Lists of locales for each platform are in ${platform}_locales.txt. List of partner channels in partner-update-channels. Generated URLs as follows:

./create_aus_urls.sh

Used test_null_updates.sh to test all snippets:

./test_null_updates.sh mu-2.0.0.14-3.0.1-null-urls.txt 2>&1 | tee before.log
# grep the log for any non-passes
grep -v '^PASS'
# no output - everything is good!

Update verify

Ran update verify on fx-{linux,mac,win32}-1.9-slave2 as follows:

mkdir -p /builds/verify/firefox-20016-301-real-major/
cd /builds/verify/firefox-20016-301-real-major/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d updates mozilla/testing/release/updates/
cvs -d cltbld@cvs.mozilla.org:/cvsroot co -d common mozilla/testing/release/common/

# get patch to trawl "Only in" directories
cd common
# on mac & linux
curl -s https://bugzilla.mozilla.org/attachment.cgi?id=304252 | patch -p0
# on win32
wget --no-check-certificate -O patch https://bugzilla.mozilla.org/attachment.cgi?id=304252
patch -p0 < patch
 cd ../updates

./verify.sh -c moz18-firefox-$platform-major.cfg 2>&1 | tee $platform.log
# had to kick along some of them because they hit slow bouncer mirrors.
# did this by killing the wget process (which makes it retry)

Add 'sl' MU snippets + null partner snippets

sl was missed in the patcher config file for the original run :(. To fix, re-ran steps 3 and 4.2 from above. Then separated out beta snippets into their own directory:

rsync aus2/ aus2.beta/
find aus2/ -type d -name "beta*" -exec rm -rfv {} \; 2>&1 | tee -a release_remove.log
find aus2.beta/ -type d -name "release*" -exec rm -rfv {} \; 2>&1 | tee -a beta_remove.log

Then synced sl snippets over to aus2-staging:

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only/
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.test/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only-test/
rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.beta/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only-beta/

rsync -e 'ssh -i /home/cltbld/.ssh/aus' -av aus2.null/ cltbld@aus2-staging.mozilla.org:/opt/aus2/snippets/staging/20080821-Firefox-MU-sl-only-null-partner-snippets/

Pushed null and test snippets:

~/bin/pushsnip 20080821-Firefox-MU-sl-only-null-partner-snippets
~/bin/pushsnip 20080821-Firefox-MU-sl-only-test

Separate ja-JP-mac snippets into a separate directory

We aren't likely going to be doing a 2.0.0.16->3.0.1 ja-JP-mac update but nonetheless we should keep these snippets around. If we end doing the update the new directories can be pushsnip'ed to enable it. test channel snippets were not separated out, but deleted from the live snippet dirs :(. If we need ja-JP-mac 2.0.0.16->3.0.1 test channel snippets we'll have to regenerate them.

# cltbld@aus2-staging.mozilla.org
cd /opt/aus2/snippets/staging
rsync -av 20080820-Firefox-2.0.0.16-3.0.1-MU 20080821-Firefox-MU-ja-JP-mac-only
rsync -av 20080820-Firefox-2.0.0.16-3.0.1-MU-beta 20080821-Firefox-MU-ja-JP-mac-only-beta
cd 20080821-Firefox-MU-ja-JP-mac-only
mv 20080820-Firefox-2.0.0.16-3.0.1-MU/Firefox .
rmdir 20080820-Firefox-2.0.0.16-3.0.1-MU
find . -mindepth 5 -maxdepth 5 | grep -v ja-JP-mac | xargs rm -rfv
cd 20080821-Firefox-MU-ja-JP-mac-only-beta
mv 20080820-Firefox-2.0.0.16-3.0.1-MU/Firefox-beta .
rmdir 20080820-Firefox-2.0.0.16-3.0.1-MU-beta
find . -mindepth 5 -maxdepth 5 | grep -v ja-JP-mac | xargs rm -rfv
# now remove ja-JP-mac from the original directories
cd ../20080820-Firefox-2.0.0.16-3.0.1-MU/
find . -maxdepth 5 -mindepth 5 | grep 'ja-JP-mac' | xargs rm -rfv
cd ../20080820-Firefox-2.0.0.16-3.0.1-MU-beta/
find . -maxdepth 5 -mindepth 5 | grep 'ja-JP-mac' | xargs rm -rfv

update-verify log

We manually scanned the update verify logs. Also compared logs with Nick's earlier update verify. Noted some differences in logs for:

sk - mac
sl - all
ru - linux, mac
ja-JP - mac
mn - win32
mb-NO - win32
nl - win32

...so ask QA to add these to the spotchecking list. Just in case.

• There is a Bulgarian (bg) locale in FF2.0.0.x, but there is no Bulgarian (bg) locale in FF3.0.x yet. This means we cannot do a major update for bg yet. I confirmed there are no update snippets for bg, so we are all ok.

Push update snippets to beta channel

First do backup:

sudo su - cltbld
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-beta
real    65m41.991s
user    1m15.712s
sys     1m9.448s

To push to beta the following directories need to be pushsnip'ed on aus2-staging:

# contains every locale except 'sl'
time ~/bin/pushsnip 20080820-Firefox-2.0.0.16-3.0.1-MU-beta
real    0m12.339s
user    0m0.013s
sys     0m0.521s
# only contains 'sl'
time ~/bin/pushsnip 20080821-Firefox-MU-sl-only-beta
real    0m0.467s
user    0m0.002s
sys     0m0.031s

Push update snippets to release channel

• In case you need to cancel/abort updates, the quickest way to do this is here

• First do backup:

sudo su - cltbld
cd /opt/aus2/snippets/staging
time ~/bin/backupsnip 20080820-Firefox-2.0.0.16-3.0.1-MU

To push to release the following directories need to be pushsnip'ed on aus2-staging:

# contains every locale except 'sl'
time ~/bin/pushsnip 20080820-Firefox-2.0.0.16-3.0.1-MU
real    0m11.777s
user    0m0.019s
sys     0m0.590s
# only contains 'sl'
time ~/bin/pushsnip 20080821-Firefox-MU-sl-only
real    0m0.500s
user    0m0.003s
sys     0m0.044s