Release Management/Chemspill

Definition

"Chemspill" is a term used to describe a security-driven rapid release.

In a "chemspill" situation we release on whichever channels necessary, with only the necessary patch(es), as fast as possible. This is usually reserved for situations where a critical security exploit is public.

Some documentation around chemspill process

• Release days for Dot releases and chemspills
• Chemspill process description. A template to copy and use for organizing incident response.
• Chemspill retrospective template. Use this for post-mortems (in draft, June 2019)
• Slides from a relman lightning talk on a chemspill in 2018
• Writing code for chemspill releases
• Security and security ratings page for reference

Past chemspills

2024 October - 0day reported by ESET

• Fixed in Firefox 131.0.2 (and Firefox ESR 128.3.1 and ESR 115.16.1) and Thunderbird 131.0.1, 128.3.1 and 115.16.0.
• bug 1923344

2024 March - pwn2own 2024

• Firefox 124.0.1 (and ESR 115.9.1)
• bug 1886842

2023 September - libvpx

• Firefox 118.0.1, Firefox ESR 115.3.1, Firefox Focus for Android 118.1, Firefox for Android 118.1, Thunderbird 115.3.1
• Bug bug 1855550
• Notes: Incident doc

2023 September - libwebp

• Firefox 117.0.1, ESR 115.2.1 & 102.15.1
• Bug: bug 1852649
• Notes: Incident doc

2022 May Pwn2Own

• Fixed in Firefox 100.0.2, Firefox for Android 100.3.0, Firefox ESR 91.9.1 (Security advisory)
• Bugs: bug 1770048 and bug 1770137
• Notes: Chemspill checklist, Whitepaper in bug 1770040

2022 March "zero days before wellness days"

• Fixed in Firefox 97.0.2, ESR 91.6.1 and later
• Bugs: bug 1758062 and bug 1758070
• Notes: Incident doc, Retrospective doc

2020 Apr

• Versions with the fix:
• Firefox 74.0.1, Firefox ESR 68.6.1, DevEdition 75.0b12 and Beta 75.0-build2 (Security advisory)
• Bugs: 1620818 and 1626728
• Notes: Incident doc; Retrospective TBD

2020 Jan "DarkHotel"

• Versions with the fix
• 8 Jan 2010: Firefox 72.0.1, 73.0b2, 74 Nightly; Firefox for Android 68.4.1, 68.5.1; ESR 68.4.1 (Sec-advisory)
• Bug(s): 1607443
• (Add geckoview based releases)
• Notes: Incident doc; Retrospective

2019 Jun "Coinbase hack"

2 chemspills during all hands work week.

• Versions:
  • Jun 18 - 67.0.3, 60.7.1esr, Firefox for Android 67.0.3. Fixed type confusion in Array.pop.
  • Jun 20 - 67.0.4, 60.7.2esr (not fennec). Fixed a sandbox escape using Prompt:Open.
• Bug(s): bug 1559845; bug 1544386; bug 1559858
• Notes: Incident doc; retrospective

2019 May "Armagadd-on 2"

Not a security breach but a rapid and focused single-issue dot release, which we treated as a chemspill in some ways. Repaired certificate chain to re-enable web extensions that had been disabled.

• Versions: 66.0.4, 60.6.2esr, 67.0b17; 66.0.5 (fennec + desktop + dev ed), 60.6.3esr, 67.0b18 (fennec + desktop + dev ed)
• Bug(s): bug 1548973,
• Notes: Incident doc; Incident closure; Technical report; EKR's Mozilla Hacks post

pwn2own 2019

IonMonkey/JIT issues

• 66.0.1, 60.6.1esr, 67.0b4
• Bugs: bug 1537924, bug 1538006
• Notes: Incident doc, retrospective

pwn2own 2018 Mar 15

Out of bounds memory write while processing Vorbis audio data.

• Versions: 59.0.1, Firefox ESR 52.7.2,
• Bugs: bug 1446062, bug 1446365
• Notes: Incident doc - Mozilla Hacks post on this chemspill

2018 Jan: Spectre/Meltdown

• Versions: 58.0.1 , 57.0.4.
• Bug(s): 1423225
• Notes: incident doc

2017 Dec: tab crash issue

Not quite a chemspill but was treated as such. Fix a crash reporting issue that inadvertently sends background tab crash reports to Mozilla without user opt-in.

• Versions: 57.0.3, 52.5.3esr, Beta 58, dev ed 58, and a system add-on to older release versions.
• Bug(s): Bug 1424373
• Notes: incident doc

2017 Mar, pwn2own

Integer overflow in createImageBitmap()

• Versions: 52.0.1, fennec 52.0.1, 52.0.1esr, 53.0b3, Fennec 53.0b3, (plus Thunderbird)
• Bug(s): bug 1348168
• Notes: checklist and planning doc

2016 Nov 30, SVG 0day

Firefox SVG Animation Remote Code Execution.

• Versions: 50.0.2, 51.0b5, and 45.5.1esr.
• Bug(s): Bug 1321066
• Notes: Planning and checklist

2016 , "Armagadd-on"

• Versions:
• Bug(s):
• Notes: https://public.etherpad-mozilla.org/p/bug-1267318

Feb 2016 Service workers issue

• Versions: 44.0.2
• Bug(s): 1245724
• Notes:

Aug 2015, pdf.js issue

• Versions: 39.0.3, 38.1.1, Firefox OS 2.2
• Bug(s): 1191284
• Advisory: https://www.mozilla.org/en-US/security/advisories/mfsa2015-78/
• Notes:

Apr 2015

• Versions: 39.0.3.
• Bug(s):
• Notes:

Mar 2015

• Versions: 36.0.3/36.04 and 31.5.2/31.5.3
• Bugs: 1144988, 1145870
• Notes: (these were at https://etherpad.mozilla.org/36-0-chemspill-Post-Mortem)