Release Automation on Trunk

Items with a strikethrough have been fixed in the staging setup.

How to run the production slaves using the staging mirrors

In order to test the tinderbox and other setup of the production slaves it is sometimes necessary to run them without touching the production servers (eg, cvs.m.o, stage.m.o, etc). Here's how:

1. Copy the staging bootstrap.cfg to the /builds/buildbot/trunk-automation-master/bootstrap.cfg on production-trunk-automation.
2. Manually sync the staging-trunk-automation cvsmirror to a clean state (trailing slashes are important!):

rsync -av /builds/cvsmirror.clean/cvsroot/ /builds/cvsmirror/cvsroot/
rsync -av /builds/cvsmirror.clean/l10n/ /builds/cvsmirror/l10n/
rsync -av /builds/cvsmirror.clean/mofo/ /builds/cvsmirror/mofo/

3. Manually clean-up staging-trunk-automation and fx-linux-1.9-slave1: On staging-trunk-automation:

rm -rf /builds/config/* /builds/tags/* /builds/release/logs/* /builds/updates/* /builds/verify/* /home/ftp/pub/firefox/nightly/2007* /home/ftp/pub/firefox/nightly/3.0b2-candidates /data/cltbld/firefox-3.0b2

On fx-linux-1.9-slave1:

rm -rf /home/ftp/pub/firefox/nightly/2007* /home/ftp/pub/firefox/nightly/3.0b2-candidates /data/cltbld/firefox-3.0b2

4. Cleanup the tagging area on production-trunk-automation

rm -rf /builds/tags/*

5. Comment out the "clean staging area" and "create staging area" BuildSteps on production-trunk-automation's master.cfg.

At this point you should be able to start a run on production-trunk-automation that doesn't touch any production servers.

General Notes

• There is a set of patches in ~cltbld/local-patches that must be applied to the /builds/cvsmirror.clean whenever it is updated from the main CVS mirror
• CVS on windows requires '-z3' or it will not work correctly. (OR follow the "msys package upgrade" steps below)
• Commented out the SYMBOL_SERVER_* and $crashreporter_* variables in mozilla/tools/tinderbox-configs/firefox/*/tinder-config.pl on the 'release' and 'l10n_release' tags.
• Bootstrap has it's own special cvsmirror in /builds/cvsmirror.bootstrap/cvsroot. master.cfg was updated to reflect this.
• Some msys packages were upgraded on trunk. Here's how to do it:

• Make sure all the tinder-config.pl's (release and l10n_release branches) have ReleaseToNightly=0 and ReleaseToDated=1 set.
• Had to install 7z on OS X slave.
• Created a moz-trunk-patcher2.cfg and checked in to local mofo mirror:

<app>
    <Firefox>
        <current-update>
            to   3.0b1
            testchannel   betatest releasetest
            <rc>
                betatest   1
                beta   1
            </rc>
            from   3.0a9
            channel   beta release
            details   http://www.mozilla.com/firefox/releases/3.0a9.html
            <complete>
                beta-url   http://staging-trunk-automation.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
                url   http://staging-trunk-automation.build.mozilla.org/?product=firefox-3.0b1-complete&os=%bouncer-platform%&lang=%locale%
                path   firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
                betatest-url   http://fx-linux-1.9-slave1.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
            </complete>
        </current-update>
        past-update   3.0a9 betatest releasetest beta release
        <release>
            <3.0a9>
                <platforms>
                    win32   2007110904
                    linux-i686   2007110903
                    mac   2007110903
                </platforms>
                <exceptions>
                    gu-IN   mac, linux-i686
                    eu   linux-i686 win32
                </exceptions>
                version   3.0a9
                completemarurl   http://staging-trunk-automation.build.mozilla.org/pub/mozilla.org/firefox/releases/3.0a9/firefox-3.0a9.%locale%.%platform%.complete.mar
                locales   be ca cs de el en-US es-AR es-ES eu fi fr fy-NL ga-IE gu-IN he ja-JP-mac ja ka ko ku lt nb-NO nl pa-IN pl ru sk sv-SE tr uk zh-CN
            </3.0a9>
            <3.0b1>
                <platforms>
                    linux-i686   2007110903
                    win32   2007110904
                    mac   2007110903
                </platforms>
                <exceptions>
                    ja-JP-mac    mac
                    ja    linux-i686, win32
                    gu-IN   linux-i686, mac
                </exceptions>
                version   3.0b1
                locales   be cs de el en-US es-ES fi fr fy-NL gu-IN ja-JP-mac ja ka ko lt nl pl ru sk sv-SE uk zh-CN
                completemarurl   http://fx-linux-1.9-slave1.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b1-candidates/rc3/firefox-3.0b1.%locale%.%platform%.complete.mar
                schema   1
                extension-version   3.0b1
            </3.0b1>
        </release>
    </Firefox>
</app>

• Had to add moz-trunk-firefox-${platform}.cfg files in mozilla/testing/release/updates. These files were empty to begin with.

Problems & Solutions

• 'Tag substep Bump died: None of ^WIN32_MODULE_PRODUCTVERSION_STRING=3.0a9pre$ found in file /builds/tags/FIREFOX_3_0_A9_RC1/cvsroot/mozilla/browser/app/module.ver: at Bootstrap/Step/Tag/Bump.pm line 149.'
  • Your cvsmirror is out-of-date. Run the Tag Builder with the cvsmirror steps enabled to resolve this. (double check this)
• Python errors on OS X (cannot import os module, etc)
  • configure seems to be finding the wrong version of Python. Worked around by moving /usr/bin/python symlink to point to /tools/python/bin/python
• CVS sometimes hangs on windows
  • Upgrade ssh or use -z3 (see above)
• Build fails on an nsinstall of scriptaculous unit tests
  • This is caused by bug 396187. Workaround by using /e/fx19rel and /e/fx19l10nrel as the tinderbox directories
    • Make sure to change bootstrap.cfg to use these new directories.
• pushDir not found at end of Build
  • last-built is not being removed at the start of a new build. remove it manually before doing a Build.
  • On Windows Tinderbox/Bootstrap seem to use different names for the build platform...'WINNT_5.2_Depend' seems to be a safe choice.
• wget errors in Repack::Execute()

wget -nv --output-document "/builds/tinderbox/Fx-Mozilla1.9-l10n-Release/Linux_2.6.18-8.el5_Depend/firefox.tar.gz" http://staging-trunk-automation.build.mozilla.org/pub/mozilla.org/firefox/nightly/3.0b2-candidates/rc1/firefox-3.0b2.en-US.linux-i686.tar.gz
End:   Mon Nov 12 10:41:20 2007
Error: wget failed or timed out.

• Make sure the HTTP server on staging-trunk-automation is running
• On Linux, this may occur because your tinder-config.pl is hardcoded to look for firefox.tar.gz. This should be updated to firefox.tar.bz2. (The '# CONFIG' lines, that is).