Test Pilot/2016Q4
From MozillaWiki
Test Pilot 2016 Q4 Objectives and Key Results
| Final Status (Dec 31) | Comments | ||
|---|---|---|---|
| Objective 1 | Grow the Test Pilot user base. | .26 | |
| KR1 | 30% improvement in Test Pilot experiment enable rate for new users | 0.50 | Starting rate is 8.02% which makes our target 10.42%. On Dec 31 the average for the quarter was 8.27% (an increase of 3.1%). However, the average for just December was 12.60% (an increase of 57.1%) so we definitely did some good, but that success is weighed down by the short period we had to measure after we pushed changes. Instead of a score of 0.03 I'd like to call this 0.5 for the positive trend going into Q1 (and a note about being able to measure within a quarter). |
| KR2 | Reduce ratio of Test Pilot users (DAU) with no experiments installed by 30% | 0.0 | At start of quarter is 57.4%, so this moves the target to 75%. Calculations averaged over the last week of the year show this ended up at 56.2%. There was probably some effect from the holidays, but we definitely weren't at 75%. |
| KR3 | At least 100 different accounts have commented on the Test Pilot Discourse forum over the course of a month. | 0.27 | We had 27 unique folks who were active in the month from mid-oct to mid-nov. |
| Objective 2 | Provide evidence that Test Pilot is useful in helping to make product decisions | .44 | |
| KR1 | Publish conclusions from three experiments and graduate them from Test Pilot | 0.33 | Universal Search |
| KR2 | Launch two new experiments | 0.0 | Closest was Cliqz, but that got bumped to Jan. Snoozetabs was functional, but quite a bit left on polish. |
| KR3 | Build an offboarding process | 1.0 | Universal Search graduated. |
| Objective 3 | Have the ability to run experiments outside of the US Market | .75 | |
| KR1 | Add localization support to Test Pilot and get 80% of users using Test Pilot in their first choice language. | 1.0 | Averaged over Q4 en-US is 85% of requests so calling this green. Note that at the end of December en-US is down to 74.71% of requests. |
| KR2 | Launch at least one locale-specific experiment in one non-English locale | 0.5 | This was Cliqz which got bumped to Jan, but due to inter-team dependencies (and, in general, Test Pilot work completed) we decided to give 0.5 here. |
Updates
- Mid-quarter review: https://discourse.mozilla-community.org/t/2016q4-okr-mid-quarter-review/12161
