User:Hoosteeno/Funnelcakes

Firefox Funnelcakes

“Funnelcakes” are special builds of Firefox that let us test changes to the acquisition funnel or product by distributing variations to real users via the download page. For example, we might change a Firefox preference to “true” that is normally “false”; we could then distribute the “true” version to a targeted subset of users and measure how that change affects Firefox use.

Anatomy of a successful Firefox install

Funnelcakes change how our download and install funnel works. Here’s how things normally go:

1. A user decides to download Firefox and finds their way to the Mozilla web site and the Firefox download page hosted there.
2. The user clicks on the “Download Firefox” button. Server technology (“the bouncer”) determines which platform, locale, channel and version the user should get, and redirects the user to that particular binary file on our CDN. The download begins.
3. When the download has finished, the user installs Firefox using the downloaded file.
4. Next, the user opens Firefox. Since this is the first time the user has opened a fresh install of Firefox, Firefox will automatically request a page served by our website, the “First Run” page.
5. The user enjoys and depends on Firefox for many years. It helps them navigate through all the times of their life.

When a funnelcake experiment is underway, the above steps are different:

• Some subset of users at step 2 download a special version of Firefox (the funnelcake version) with a unique anonymous number associated with the entire experiment (funnelcake id).
• When those funnelcake users visit the First Run page in step 4, the URL for that page includes a parameter (usually ?f=funnelcake-id) identifying them as part of the cohort. This allows us to understand how the funnelcake affected install rates.
• When the user is enjoying Firefox, the funnelcake ID is also included in the distribution.id field of Firefox. This allows us to understand usage and retention rates of those anonymous cohorts.

How to run a Funnelcake experiment

TLDR; Design an experiment, ask the Release Engineering team to build a special version of Firefox, and ask the Mozilla.org team to distribute that special version to a particular segment of downloaders.

More specifically:

1. At least 4 weeks^* before you hope to launch your experiment, design it. A good experiment will have a hypothesis, a measurement method, a timeline, and any details someone else might require to reconstruct it at a later date. The Firefox Growth team has experiment templates for Google Docs and Mana. Here is an example funnelcake experiment. You can share your experiment by adding it to this list of experiments.
2. Convince stakeholders on teams involved that the experiment is worthwhile. Talk to the Growth Team if you want help assessing this.
3. At least 2 weeks^* before you hope to launch your experiment,
   1. File a bug for the release engineering team explaining what the funnelcake should include, which version and channel of Firefox it should be based on, what locale it is for, and when distribution should begin. Here is an example RE bug.
   2. Add your funnelcake to the big list of all funnelcakes ever and fill in all the fields that spreadsheet requires. This is also where you’ll get your funnelcake number.
   3. File a bug for the Mozilla.org team to build an Optimizely experiment to distribute the funnelcake to a specific segment (include relevant factors such as locale, geography, user agent, segment size), as well as any other changes. Here is an example Mozilla.org bug.
   4. If the experiment is sufficiently complex to involve many more bugs than that, file a meta bug to organize all the other bugs. (In many cases the two bugs above are sufficent).
4. Participate in the process however you can: Testing, building Optimizely experiments, following up on bugs, making sure the launch is smooth, etc.

* Add more time if your experiment is very complex, e.g. requires substantial changes to the product or download funnel.

FAQ

Q: How many users do I need for statistical validity? A: It depends on your needs, but a common rule of thumb is you need 40,000-50,000 installs (which implies a larger number of downloads -- determined by the conversion rate of the segment you’re targeting). Use a tool like Optimizely’s sample size calculator to plan your experiment.

Q: Is there a list of all the funnelcakes ever run? A: Here is a list of all the funnelcakes ever run!

Q: How are numbers for funnelcakes assigned? A: They are arbitrary (they don’t have any algorithmic meaning), but sequential. You can see the sequence on the big list of funnelcakes.

Q: How do we measure retention using funnelcakes? A: One field in the big list of funnelcakes is “distribution ID”. This field is included in the channel that Firefox sends to telemetry. We can monitor telemetry data (including retention) for users of a particular channel (such as a funnelcake) using tools like this.

Q: How do we use funnelcakes to gather qualitative data? A: The firefox user advocacy and research teams can target specific distribution IDs (such as the one assigned to a funnelcake) with heartbeat surveys. Furthermore, we can distribute funnelcakes to usertesting.com testers.