Build config/Projects
This page enumerates active and potential projects for the Build Config module. This module encompasses the core build system (read: in-tree code). Although, some projects do involve automation.
Please add new projects to the Proposed Projects section at the bottom of the page.
Contents
- 1 Accepted Projects
- 1.1 MozillaBuild
- 1.2 Automation Interaction and Needs
- 1.3 Core Build System
- 1.3.1 Move all build config for linking libxul to moz.build
- 1.3.2 Improve chrome building, packaging
- 1.3.3 Minimize ICU impact
- 1.3.4 Improve SpiderMonkey build system integration
- 1.3.5 Fix the Fennec build system
- 1.3.6 Use nose for running Python tests
- 1.3.7 Kill client.mk
- 1.3.8 Tup support
- 1.3.9 reinvent mozconfigs
- 1.4 mach
- 1.5 Developer workflow
- 2 Proposed Projects
Accepted Projects
MozillaBuild
MozillaBuild is the Windows development environment. We have a few projects for it.
Release 1.9.0
MozillaBuild 1.9.0 with GNU Make 4 should be released.
Tracked by bug 928594.
Overhaul to Use msys2
- MozillaBuild 2.0* will be an overhaul of MozillaBuild to use msys2, a decent terminal, and a bunch of other improvements.
See https://groups.google.com/d/msg/mozilla.dev.builds/X760-Y4DSRc/LTlo39uNKSEJ
Automation Interaction and Needs
These are projects that relate to how the core build system interacts with automation.
Establish Clear API for Command Interaction
We want all in-tree calls in automation to be proxied through well-defined points so we have a clearly supported API.
bug 978211 tracks.
Build with GNU Make 4.0 on Windows
Automation should use GNU Make 4.0 on Windows.
bug 927672 tracks.
A follow-up goal is likely to remove code for supporting pymake and eventually pymake itself from the tree.
Cross-compile Mac builds from Linux
This is a request from Release Engineering. For capacity reasons, we want to perform Mac builds on Linux. We should add support for this.
Core Build System
Move all build config for linking libxul to moz.build
This project involves making it possible for moz.build processing to emit the set of processes and dependencies needed for linking libxul. We need to move config.mk and rules.mk logic to Python. We need to support rules for generating code. At the end of the day, we should be able to emit a single make file containing every compiler and linker command needed to produce libxul. This work will enable Tup, Ninja, etc building.
Improve chrome building, packaging
We have a number of build actions that run unconditionally, either due to clownshoes (like removing large parts of dist/) or due to bad build rules. These build actions add excessive overhead to light builds. Light builds are so slow that developers commonly perform a partial tree build and hope they build everything they needed to. This is playing with fire and this practice needs to be curtailed through us providing faster light builds.
This project is about minimizing the amount of unnecessary build actions we perform.
Specific actions in this goal include:
- Tracking more files in install manifests
* Finish moving all mochitests to manifests * Add JS files to manifests (likely via moz.build porting)
- Conditionally processing jar.mn files
- Handle headers from external projects better
- Handle XPCOM manifest files more efficiently (likely through some moz.build powered mechansim)
- Proper dependencies for Preprocessor.py
- Capture INSTALL_TARGETS usage in moz.build
Progress towards this goal is measured by comparing the number of non-make processes launched during no-op and light builds over time. This number needs to go down. Drastically.
Minimize ICU impact
ICU has been a world of pain for the build system. It makes builds significantly slower and it is often the long pole, especially for builds close to the no-op side of the spectrum.
We need to mitigate the impact of ICU on the build system. This might mean loudly recommending people not build it (although more and more features want to start using it, so likely not a long term solution). This might mean someone contributing patches back to ICU to make its build system suck less.
Improve SpiderMonkey build system integration
The way SpiderMonkey integrates with the build system is very hacky. Nobody likes it - including the SpiderMonkey developers. Furthermore, the way things are slows down builds. e.g. SpiderMonkey has its own configure script, which is extremely similar to the one in m-c. This slows down configure time, especially on Windows. The goal here is to integrate SpiderMonkey into the main build system. We'll also provide mach command(s) for building just SpiderMonkey (they've asked me for this).
Fix the Fennec build system
The Android make rules are horrible. They are slow and do a lot of actions when not needed. There is much room for improvement here.
Use nose for running Python tests
Our current Python unit testing solution is quite fragile. Failures will result in all tests being aborted. We don't get as much context from failures as we'd like. It's somewhat difficult to run specific tests in isolation (not whole test files).
The nose package has solved many of these problems and integrating it into the tree would make a lot of the issues go away. This will likely involve defining Python unit tests in moz.build files somehow. We can probably define things as a directory containing test files plus an optional regex used to discover them. Generally speaking, nose's algorithm for finding tests is pretty robust. Worst case, we can adapt our existing tests that don't conform to its expectations. This is probably a good idea anyway.
Kill client.mk
client.mk is a glorified shell script. The way it interacts with the world (especially mozconfigs) is very hacky. client.mk's days are numbered. Everything it does should be replaced with mach commands.
This requires release automation to not use client.mk (another project).
Tup support
We want to be able to build the tree with Tup, not make. Benefits: Faster and proper builds. Eliminate clobbers.
This requires more data to live in moz.build.
We would support this as a local development mode first then move to automation when it is stable.
reinvent mozconfigs
There is a longstanding desire for mach to support config files and for some aspects of mozconfigs to be reinvented. These can be independent, but they may overlap.
mozconfigs are susceptible to many of the same problems as Makefiles: you can do anything in them and actions often don't have an effect. We want to be more strict about mozconfigs so people can be sure their content therein actually does something. This especially holds true for the variables exported to client.mk via mk_add_options. At the very least, we should likely kill mk_add_options and replace it with something saner. At the most, we should kill mozconfigs as shell scripts and replace them with something else (possibly sandboxed Python, like moz.build files).
mach needs config files. There are all kinds of options that are begging for user preferences. e.g. display of the build footer, whether to colorize output, and whether to prefix output lines with times. These are all currently behind environment variables or worse - command line arguments. Using command line arguments to control UI behavior is very silly and not very user friendly. mach needs config files.
A big question is whether mach config files and mozconfigs should blend together. Theoretically, mach is a standalone and generic CLI tool. However, that shouldn't prevent us from stuffing build config data into mach config files. Another big question is what should mach config files look like? .ini is a popular choice. There are even dialects of .ini that support variable interpolation. We have configobj (http://www.voidspace.org.uk/python/configobj.html) in the tree for a reason. Another option is sandboxed Python. Traditionally, config files are static, so .ini seems like the better fit. However, if we were to put the mozconfig equivalent in mach config files, people tend to utilize the power of shell scripts for branching, etc, so a static .ini format would have hard to support. Perhaps the mach.ini file contains a "[build] config_file =" entry that points to a script to execute. As you can tell, there's lots of thinking and bikeshedding involved in this goal. It shouldn't be undertaken lightly. That being said, simple mach.ini files without regard for build config/mozconfigs can probably be done with little thought as long as we have an escape plan.
mach
Config file support
There are many options being landed in mach commands that are effectively user preferences. There is no good way to define sticky defaults for those options. mach needs config files so people can tailor their environments to their liking. We should add config file support to mach.
Faster startup
Mach has become slower as more commands are added. We should look at making it faster.
More robust available command support
Support for detecting what commands are runnable at run-time has been added recently. When you run |mach help| it even prunes the list to commands that should work. We should hook up most of the commands to this mechanism so people can't do things like attempt to run tests when the tree isn't built.
Improve comm-central and fxos integration
Mach integration with comm-central and b2g is fragile. We should make that work so we don't have to continue supporting legacy interfaces and so developers there don't complain so much about not being able to use mach. This is more important than ever since functionality like test running is starting to require mach.
Developer workflow
Ability to develop browser/Firefox without compiling Gecko
In theory we can provide a build mode so developers can build just the browser app without having to build all of Gecko/libxul. This will dramatically reduce the horsepower required to obtain a working build from source. The build configuration would likely download a known working binary distribution of Gecko (possibly a xulrunner build). This is similar to how some Firefox OS development works with Gaia using a pre-built Gecko.
Benefits: Firefox frontend developers are happier. Retains more community contributors due to easier contribution?
Build with containers, Docker
Release engineering is now publishing archives of their chroot environments at https://s3.amazonaws.com/mozilla-releng-mock-archive/mock_archives.json. It would be rad if someone hooked this up to mach and the build system so local builds could be performed in containers using the exact same bits that automation uses.
"Metrics" for build system and command usage
It's always nice to make data-driven decisions. Currently, we have no reliable way of knowing what hardware people are building on, what mach commands they use, which parts of the tree are getting built the most locally, etc.
We would like to instrument the build system and build tools to record and report usage info to Mozilla so we have a better understanding of how people are doing things. This will enable us to make better decisions about priorities and should result in a better build experience and higher productivity.
It is relatively easy to stand up a server to collect this data. The volume of data is probably low enough that we only need a SQL server on a single machine somewhere (at least if release automation machines are excluded).
While other companies that do this kind of instrumentation can get away with having it enabled by default because all developers are under the corporate banner and thus not susceptible to privacy concerns because they signed them away as part of employment, Mozilla can't. Anyone can download the source code and we can't have things reporting automatically without notification. It's likely not legal and it's definitely not the Mozilla way. However, for this data to be useful and not skewed, we'd want near 100% participation. We obviously need to work with privacy and legal to flush out how this would work.
Proposed Projects
Please place new project ideas here.
