Writing Code for Chemspill Releases
Writing code for a chemspill release is different from writing most other code in Firefox. Chemspill releases are what we do to handle a security vulnerability that's publicly known or being used in the wild. The goal of these releases is to ship very quickly, hopefully in less than 24 hours, from when we learn about the problem. Rollout of these releases is not throttled. So we're shipping code really fast. The code goes to millions of users within hours after release, and very few safeguards exist. There's no chance for users to report bugs, or for us to triage incoming crash reports, or to look at telemetry.
We've messed these up in the past, to the point of having to ship more dot releases to fix the problems. We've also almost messed up in ways that would have made us do this. We haven't *really* messed up, though, by breaking updates, or breaking the ability to start Firefox and thus update.
When you're writing code for a chemspill release:
- "probably" isn't good enough. "sounds like it should work" is not good enough. Can you prove it? Go through the things that could possibly go wrong.
- change as little as possible (effects, not diffs). Don't change dependencies, and don't change web-exposed behavior more than absolutely required.
- Defense in depth (multiple fixes that would fix the issue) may be a good idea. But be aware that some of the fixes might be too risky for a chemspill release, and it may be worth getting the others out sooner. Falling back to a controlled crash may be better than leaving part of the security bug unpatched.
When you're reviewing code for a chemspill release (which the author of the code should do too):
- go through all the possible codepaths and understand all of the cases (and do this on all relevant branches, not just mozilla-central)
- a code reviewer should probably rewrite the patch from scratch; you might find interesting things in the differences between the patches.
