Intro

Firefox and Thunderbird releases are currently done using the Bootstrap automation scripts, which call into Tinderbox client to do the actual build.

Buildbot calls bootstrap, parallelizing and serializing where needed.

Bootstrap

Bootstrap is a simple Perl framework intended to take the formerly manual release process and automate it, with as little change to the process as possible.

Buildbot

Buildbot is a continuous integration tool, similar to Tinderbox but written in Python as a client/server Twisted application.

We have a vendor branch in mozilla/tools/buildbot, based on Buildbot's 0.7.5 release.

Mozilla-specific Buildbot install instructions

Buildbot user manual

Current status

This system is used for all current Firefox branches. Public documentation for each release is posted under the "Build" links on the Releases page.

All configs are checked into CVS under mozilla/tools/buildbot-configs/automation.

Roles and resource requirements

• buildbot master
  • keeps logs, manages overall process
• ftp/stage.m.o
  • fileserver, both public and private areas
  • FTP candidates - 20GB storage
  • e.g. stage:/home/ftp/pub/firefox/nightly/2.0.0.4-candidates/
  • FTP private staging - 20GB storage
    • e.g. stage:firefox-2.0.0.4/
  • FTP release - 6GB storage
    • e.g. stage:/home/ftp/pub/firefox/releases/2.0.0.4/
• "tagging" builder
  • checks out source and applies tag
  • 2GB storage
    • e.g. karma:/builds/tags/FIREFOX_2_0_0_4_RELEASE/
• "source archive" builder
  • builds source archive and pushes for QA
• "linux/mac/win32 firefox builders"
  • builds firefox and pushes for QA
  • l10n/update verification
  • needs 2GB memory, 10GB storage (each)
    • e.g. prometheus-vm:/builds/tinderbox/Fx-Mozilla1.8-Release/
• "updates builder"
  • downloads and inventories a set of complete firefox updates, generates partial updates, creates AUS configuration ("snippets")
  • updates - 1GB memory, 5GB storage
  • e.g. prometheus-vm:/builds/updates/firefox-2.0.0.4/
• "stage builder"
  • creates private staging area on FTP, renames files for release
  • see "fileserver" requirements, above
• Automatic Update Server (AUS), aus2.m.o
  • 10GB for config files, backups and staging area
  • e.g. /opt/aus2/incoming/3/Firefox/2.0.0.4/, /opt/aus2/snippets/staging/20070523-Fx-2.0.0.4/, /opt/aus2/snippets/backup/20070611-1-pre-20070611-Fx-2.0.0.4.tar.bz2

Updating Staging release version

• Bump config versions in mozilla/tools/release/Makefile, mozilla/tools/configs/fx-moz18-staging-bootstrap.cfg, mozilla/tools/buildbot-configs/automation/staging/master.cfg e.g. bug 407672

• As cltbld@staging-build-console, disable cltbld's nightly cronjob to prevent accidently starting mid-way through update.

• As cltbld@staging-build-console, refresh cvsmirror by doing:

cd /home/cltbld/mozilla/tools/release
cvs up
export CVS_RSH="/home/cltbld/ssh_prod.sh"
make cvsmirror

• Update buildbot configs. These are symlinked from the buildbot-configs checkout (of mozilla/tools/buildbot-configs/automation/staging/).

As buildmaster@staging-build-console:

# pull configs and restart buildbot
cd /home/buildmaster/TestBot
buildbot stop `pwd`
cd buildbot-configs && cvs up && cd ../
buildbot start `pwd`

As cltbld@staging-build-console:

# update mofo scripts
cd /data/cltbld/bin
cvs up

• Remove and re-checkout tinderbox-configs directories on build machines. Since the mirror takes checkins, the local checkout thinks that it has a higher revision and won't update to the now-refreshed mirror.

# e.g. linux
cd /builds/tinderbox/Fx-Mozilla1.8-release
rm -rf tinderbox-configs
cvs -d :ext:cltbld@staging-build-console.build.mozilla.org:/builds/cvsmirror/cvsroot co -d tinderbox-configs -r MOZILLA_1_8_BRANCH_release mozilla/tools/tinderbox-configs/firefox/linux

For l10n use the MOZILLA_1_8_BRANCH_l10n_release branch.

• re-enable cltbld's nightly cronjob

Setup notes (historical)

• Notes on Staging Setup
• Notes on Production Setup
• Notes on Trunk Automation Setup