Overview

Shield Studies are controlled A/B tests deployed as part of the Shield project. These studies allow us to compare proposed changes to the default experience in Firefox for representative populations before shipping those changes to everyone.

Shield Studies are available on all channels. Individual studies can be opt-out or opt-in depending on the nature of the study and the data being collected. Opt-out Shield Studies can only collect Type 1 and 2 data. This is the same type of basic interaction data we collect by default as defined in our privacy policy. There may be instances where we want to collect data that is not covered by our default data collection policies. In those instances you will be prompted to opt-in to the study. A complete description of the study and any additional data that would be collected will be disclosed before enrolling. A great example of this is Pioneer, which is actually an opt-in Shield Study currently in the wild.

At the end of the study period, the study will expire and return your system to the previous state. When the study expires, you may be asked to fill out a survey based on their experience.

Guiding Principles

All Shield studies must adhere to a common set of principles that govern the program. The current list of guiding principles are as follows:

• All Shield Studies must be designed to answer a specific question - Shield is a tool for evaluating ideas and features in the product. If you are not trying to evaluate a proposed feature or idea there are other means of shipping your code.
• Shield studies will always respect user privacy in accordance with our data collection policies
• All Shield studies will adhere to the scientific method for answering complex questions - Generate a hypothesis, test, collect and analyze data, validate or refute hypothesis, refine, repeat..
• All Shield studies require a PHD (Product Hypothesis Doc) - A PHD or Product Hypothesis Doc outlines the overarching research question the study is trying to answer. It requires the study author to think critically about the problem and the outcomes long before the study ships.
• All Shield studies must named accurately - The name of the study should reflect the general theme of the study and help assure our users that the study is legitimate.

Shield Studies FAQ

Who Approves a Shield Study before it ships?

Add-on based Shield Studies must be approved by:

• A Firefox Peer
• QA
• Release Management
• A Data Steward
• A member of the core Shield Team

Preference Experiments must be approved by:

• QA
• Release Management
• A member of the core Shield Team

Additional approvals may be required by:

• A Firefox Executive
• Legal
• Others

Where Are Approvals Tracked?

All approvals must be made explicitly by the signers above in the launch bug.

What data do Shield Studies normally collect?

By default, Shield Studies will collect this data:

• Firefox was used on a day
• Usage hours
• Study info: Study Name/ID, Experimental Branch
• Answers provided in followup survey
• Study status transition events

Mechanism:

- at ENTER, EXIT, INSTALLED, INELIGIBLE, EXPIRED, USER_DISABLE, ENDED_NEGATIVE, ENDED_NEUTRAL, or ENDED_POSITIVE events, - send a `shield-study` packet containing the Unified Telemetry Environment.

Additional measures may be added to individual studies pending approval from a data steward. All data collected will be disclosed prominently to the user at the time of consent.

Are Shield Studies just Add-ons?

Shield Studies are a special type of restartless extension. They can do anything a regular restartless extension can do.

Do Shield Studies need to be signed?

Yes! Shield Studies need to be signed to run in FF 42+, just like all other add-ons.

How long do Shield Studies last?

Shield Studies live for a fixed period and are uninstalled by the system at the end date or after the specified elapsed time. Usually that time is 14 days from the install date, but studies can last much longer if needed.

What are participation rates for opt-in Shield Studies?

Participation for opt-in Shield Studies is currently around 1-2% of randomly selected participants. We are working on improving the overall flow.