Original Post

We’ve been using a fork of the Fedora Project’s Mock tool to produce b2g builds for a couple months now. I described this tool in an earlier post. This post is about how we are using this fork at Mozilla.

Integrating with Buildbot automation

Mock integrates with Buildbot through the MockInit, MockInstall, MockCommand and MockProperty classes. The first two commands are used to configure and create the environment while the third and fourth are used to execute commands inside of a build environment.

MockInit is a basic wrapper for calling mock --init -r target_name. It is provided to encapsulate the logic for this relatively simple operation. Internally, initializing an environment means making sure that the environment is completely clean and has been recreated based on the contents of the root cache and the base packages specified in the mock config file.

MockInstall is also another basic wrapper. MockInstall internally calls mock -r target_name --install package1 package2 packageN. This step will install into the specified target environment the packages needed and any the packages requested depend on. This requires all build tools and dependencies be packaged and added to a Yum repository. Doing this is highly desirable regardless of whether or not Mock is being used.

MockCommand is derived from the standard ShellCommand as intended to operate like the base class. When in non-Mock mode (i.e. use_mock is set to False), this command does nothing to the configuration options and passes all calls straight to the base ShellCommand. When in Mock mode, the command runs its magic method to modify the command line to properly pass the requested buildbot environment through to the Mock call as well as passing the correctly modified command line given the working directory requested (using build environment paths) and the target that should be used.

MockProperty is to SetProperty what MockCommand is to ShellCommand with the important distinction that MockProperty inherits singly from MockCommand and duplicates logic from SetProperty.

In the current implementation, Mock has /builds/ bind mounted into the build environment. This means that all paths under /builds/ have the exact same paths in the build environment and on the build host. If the build process is substantially reworked to use a scripting framework other than Buildbot, it might make sense to stop making the /builds/ directory available inside of the build root. All commands which are “build system” commands are run under a MockCommand with use_mock set to true. Currently, there are no uploads or X11 dependent jobs run under mock. I have a blog post that details X11 access and uploads could be fixed by bind-mounting the host’s ~/.ssh directory into the build environment.

RPM and Yum infrastructure

Because all dependencies are fetched and installed through Yum+RPM on each build, it is necessary to package all tools as RPMs. How RPMs are built and [Yum repositories are created] is outside the scope of this document. The current setup is for Mozilla to internally mirror the host OS and the build environment OS as well as a custom repo for each distribution. Ideally, the mirrored copy of the repositories would be completely unaltered. I haven’t experimented with overriding core packages yet, but we should be fine to put newer packages that override OS packages in the custom Mozilla repository. As for building RPMs, I’ve been using spec files in my repository to build the rpms.

This repository has a directory for each distributions for which custom packages are generated, currently only ‘centos6′. Inside the distribution directory are directories for each package. There are no directories for each architecture, since rpm has facilities for specifying which architectures are valid for a given package. Each package directory contains small patches and source files needed to build the package, a tooltool manifest for larger files and a symlink to a file called “actions.sh”. This is a symlink to a script in a hidden directory at the root of the repository. In this script, there are commands for the following actions:

   fetch — use tooltool to fetch the sources
   store — add the specified file to the tooltool manifest
   build — perform a simple rpmbuild on the given spec file
   mock_build — perform a mock build of the package using upstream’s mock package

It is the responsibility of the package author to add sources, specs and manifests to the repository and upload the files to the tooltool resource host. The mock_build command can be invoked with the syntax ./actions.sh mock_build fedora-16-x86_64 x86_64 my_package.spec. The SRPMs created should be included in the list of rpm files included in Yum repository creation, since they are the canonical source of what the binary packages were built from.

Creating a new target

In the case that a new build environment configuration is needed, I’d recommend starting with an existing mock_mozilla configuration. The values that you’re likely going to be working with are the root name, arch list, base package list, bind mount and repository list. Deploying the cfg file is required and should be done through something like puppet.

Development

A release of mock_mozilla can be generated by running

git clone git://github.com/jhford/mock_mozilla.git
cd mock_mozilla
./autogen.sh
./configure
sudo make install
make srpm # this is to get the tarball, don't care about the srpm created here
mock -r epel-6-x86_64 --buildsrpm --spec mock_mozilla.spec --sources . --resultdir srpm --target x86_64
mock -r epel-6-x86_64 srpm/mock_mozilla-1.0.1-1.el6.src.rpm --resultdir x86_64 --target x86_64

The code to mock_mozilla is on github.