Fennec issues

Build 1

Release sanity failed for Fennec due to an issue with the ordering in the mozconfig (bug 1042078). We hit this in 31.0b1, but unfortunately the fix wasn't ported to nightly/aurora, so we hit it here again.

release not started

   2014-07-22 08:05:14-0700 [Broker,124058,10.26.48.51] perspective_addChange called
   2014-07-22 08:05:14-0700 [Broker,124058,10.26.48.51] adding change, who lmandel@mozilla.com, 1 files, rev=None, branch=releases/mozilla-beta, repository=, comments , category None, project
   2014-07-22 08:05:14-0700 [Broker,124058,10.26.48.51] baseTag 'FENNEC_31_0b11_' IS NOT in script_repo_revision 'FENNEC_32_0b1_RELEASE'
   2014-07-22 08:05:14-0700 [Broker,124058,10.26.48.51] baseTag 'FENNEC_31_0b11_' IS NOT in script_repo_revision 'FENNEC_32_0b1_RELEASE'
   2014-07-22 08:05:14-0700 [Broker,124058,10.26.48.51] baseTag 'FENNEC_31_0b11_' IS NOT in script_repo_revision 'FENNEC_32_0b1_RELEASE'
   2014-07-22 08:05:14-0700 [Broker,124058,10.26.48.51] baseTag 'FENNEC_31_0b11_' IS NOT in script_repo_revision 'FENNEC_32_0b1_RELEASE'

<rail> looks like bm81 ignored the sendchange because it either had not up to date checkout or the reconfig went flaky

Another reconfig happened at 9:10 PDT (and fixed the missing tag). The following sendchange has been manually resent:

   buildbot sendchange --username lmandel@mozilla.com --master buildbot-master81.build.mozilla.org:9301 --branch releases/mozilla-beta -p products:fennec -p script_repo_revision:FENNEC_32_0b1_RELEASE release_build

change sent successfully

Firefox issues

Build 1

Release sanity failed for Firefox due to a missing locale in shipped-locales (bug 1042054). Once the l10n folks fixed that, it got kicked off fine.

Fix up from failed tagging

Tagging timed out because hg is slow. It was most of the way through the job, so I kicked the rest by hand. I did the locale repos as ffxbld on b-linux64-hp-0034.build.releng.scl3.mozilla.com:

mkdir /builds/slave/bhearsum-32.0b1
cd /builds/slave/bhearsum-32.0b1
for i in vi xh zh-CN zh-TW zu; do hg clone -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' ssh://hg.mozilla.org/releases/l10n/mozilla-beta/$i; done
cd vi
hg up -r e5e5cef6c0ee
hg up GECKO320b1_2014072210_RELBRANCH
hg tag -u ffxbld -r e5e5cef6c0ee -m "Added FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1 tag(s) for changeset e5e5cef6c0ee. DONTBUILD CLOSED TREE a=release" -f FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1
hg push -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa'
cd ../xh
hg up -r 7d88d1363dab
hg branch GECKO320b1_2014072210_RELBRANCH
hg tag -u ffxbld -r 7d88d1363dab -m "Added FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1 tag(s) for changeset 7d88d1363dab. DONTBUILD CLOSED TREE a=release" -f FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1
hg push -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' --new-branch
cd ../zh-CN
hg up -r 0b9964edf617
hg branch GECKO320b1_2014072210_RELBRANCH
hg tag -u ffxbld -r 0b9964edf617 -m "Added FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1 tag(s) for changeset 0b9964edf617. DONTBUILD CLOSED TREE a=release" -f FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1
hg push -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' --new-branch
cd ../zh-TW
hg up -r c5333e78fa1d
hg branch GECKO320b1_2014072210_RELBRANCH
hg tag -u ffxbld -r c5333e78fa1d -m "Added FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1 tag(s) for changeset c5333e78fa1d. DONTBUILD CLOSED TREE a=release" -f FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1
hg push -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' --new-branch
cd ../zu
hg up -r f3aac54b9951
hg branch GECKO320b1_2014072210_RELBRANCH
hg tag -u ffxbld -r f3aac54b9951 -m "Added FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1 tag(s) for changeset f3aac54b9951. DONTBUILD CLOSED TREE a=release" -f FIREFOX_32_0b1_RELEASE FIREFOX_32_0b1_BUILD1
hg push -e 'ssh -l ffxbld -i ~/.ssh/ffxbld_dsa' --new-branch

And the build repos (mozharness, compare-locales, partner-repacks, buildbot) as me on my laptop. I just added the FIREFOX_32_0b1_{RELEASE,BUILD1} tags to their production branches and pushed them.

Once all of that was done I had to get the automation going again. Because the builders that are downstream of tagging are triggered by a Dependent scheduler, there's no way to get them going by poking the database. To do it I had to add skip_tag to the firefox release config on the production branch (https://hg.mozilla.org/build/buildbot-configs/rev/c2a79709b877), reconfig the build+scheduler masters, and rerun the sendchange:

# cltbld@bm81
cd /builds/releaserunner
bin/buildbot sendchange --who lmandel@mozilla.com --master buildbot-master81.build.mozilla.org:9301 --branch releases/mozilla-beta -p products:firefox -p script_repo_revision:FIREFOX_32_0b1_RELEASE release_build

Then I backed out my change, to make absolutely sure it won't interfere with more releases.

missing tags

parner_repacks on linux64 and macosx64 failed because, FIREFOX_32_0b1_RELEASE tag was not present in https://hg.mozilla.org/build/partner-repacks. Added tag and re-triggered failed builds.

failed updates

info.txt files are generated during the build but they are not available on:

http://stage.mozilla.org//pub/mozilla.org/firefox/nightly/32.0b1-candidates

and wget fails with the following error:

  wget of http://stage.mozilla.org//pub/mozilla.org/firefox/nightly/32.0b1-candidates/build1/linux_info.txt failed. at tools/lib/perl/MozBuild/Util.pm line 603.

We also didn't get any partial updates uploaded. Turns out to be fallout from a build system changed, filed as bug 1042432. We reverted part of the build change, which was for the s3 caching that we don't use on beta or higher. Started build 2 with that.

Build 2

push to mirrors

push to mirrors failed because http://stage.mozilla.org/pub/mozilla.org/firefox/releases/32.0b1/ it's already there (created by build1)

ssh -l ffxbld -i /home/cltbld/.ssh/ffxbld_ds stage.mozilla.org \
test ! -d /pub/mozilla.org/firefox/releases/32.0b1/ 
returned non-zero exit status 1

The problem is described here in the wiki: https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Troubleshooting#Overwriting_files_that_have_been_pushed_to_releases.2F

filed bug 1042847: https://bugzilla.mozilla.org/show_bug.cgi?id=1042847
removed /pub/mozilla.org/firefox/releases/32.0b1/ directory from stage.mozilla.org

no updates from previous RC

ran the following (thanks rail):

   # ffxbld@aus3-staging
   cd /opt/aus2/snippets/staging
    
    
   # test updates, using 31.0b7 snippets
    
   buildid_31b7_bld1=20140703154127
    
   buildid_31RC_bld1=20140714151536
   buildid_31RC_bld2=20140716183446
    
   mkdir -p Firefox-32.0b1-build2-rc-updates-test/Firefox/31.0/{Darwin_x86-gcc3-u-i386-x86_64,Darwin_x86_64-gcc3-u-i386-x86_64,Linux_x86-gcc3,Linux_x86_64-gcc3,WINNT_x86-msvc}
   for i in Darwin_x86-gcc3-u-i386-x86_64 Darwin_x86_64-gcc3-u-i386-x86_64 Linux_x86-gcc3 Linux_x86_64-gcc3 WINNT_x86-msvc; do
     rsync -av /opt/aus2/snippets/pushed/Firefox-32.0b1-build2-test/Firefox/31.0/$i/$buildid_31b7_bld1/ Firefox-32.0b1-build2-rc-updates-test/Firefox/31.0/$i/$buildid_31RC_bld1/
     rsync -av /opt/aus2/snippets/pushed/Firefox-32.0b1-build2-test/Firefox/31.0/$i/$buildid_31b7_bld1/ Firefox-32.0b1-build2-rc-updates-test/Firefox/31.0/$i/$buildid_31RC_bld2/
   done
   # push test live, double check that the directory doesn't have "release" directory
   pushsnip Firefox-32.0b1-build2-rc-updates-test

... and the following after the push of the final snippets

  # ffxbld@aus3-staging
  cd /opt/aus2/snippets/staging
   
   
  # test updates, using 31.0b7 snippets
   
  buildid_31b7_bld1=20140703154127
   
  buildid_31RC_bld1=20140714151536
  buildid_31RC_bld2=20140716183446

   # live updates
   mkdir -p Firefox-32.0b1-build2-rc-updates/Firefox/31.0/{Darwin_x86-gcc3-u-i386-x86_64,Darwin_x86_64-gcc3-u-i386-x86_64,Linux_x86-gcc3,Linux_x86_64-gcc3,WINNT_x86-msvc}
   for i in Darwin_x86-gcc3-u-i386-x86_64 Darwin_x86_64-gcc3-u-i386-x86_64 Linux_x86-gcc3 Linux_x86_64-gcc3 WINNT_x86-msvc; do
     rsync -av /opt/aus2/snippets/pushed/Firefox-32.0b1-build2/Firefox/31.0/$i/$buildid_31b7_bld1/ Firefox-32.0b1-build2-rc-updates/Firefox/31.0/$i/$buildid_31RC_bld1/
     rsync -av /opt/aus2/snippets/pushed/Firefox-32.0b1-build2/Firefox/31.0/$i/$buildid_31b7_bld1/ Firefox-32.0b1-build2-rc-updates/Firefox/31.0/$i/$buildid_31RC_bld2/
   done
    
   pushsnip Firefox-32.0b1-build2-rc-updates

Releases/Firefox 32.0b1/BuildNotes

Contents

Fennec issues

Build 1

release not started

Firefox issues

Build 1

Fix up from failed tagging

missing tags

failed updates

Build 2

push to mirrors

no updates from previous RC

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

How to Contribute

MozillaWiki

Around Mozilla

Tools