Leanplum Contextual Hints
Test Plan (iOS & Android) - July 2017
- Goal:
With the initial implementation of the Leanplum SDK into our mobile browser products, we want to start A/B testing contextual hints to increase retention.
- Overarching Hypothesis:
We believe that by promoting contextual hints that helps users explore and change settings within the browser, we will drive DAUs and retention.
iOS
- iOS Hypothesis 1:
We believe that by promoting Focus to our privacy conscious users inside Firefox, we will increase installs and DAUs in Focus, and increase retention on both apps (We know that users who have Focus and Firefox installed retain better (Adjust)).
- iOS Hypothesis 2:
We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.
- iOS Hypothesis 3:
We believe that by promoting tips and tricks such as customizing settings to our users inside Firefox, we will increase DAUs and retention.
- iOS Hypothesis 4:
We believe that by promoting Pocket to our users inside Firefox, we will increase overall DAUs and retention of the Pocket app.
- iOS Hypothesis 5:
We believe that by promoting Pocket Stories to our users on Firefox, we will increase overall DAUs and retention of the Firefox browser.
- iOS Hypothesis 6:
We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.
| Hyp. # | Message Name | Trigger | Targets |
|---|---|---|---|
| H1 | Opened New Tab - Focus Promotion | Open new tab | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H2 | Open Bookmark - Sync Promotion | Open bookmark | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H2 | Save Bookmark - Sync Promotion | Save bookmark | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H2 | Save Login and Password - Sync Promotion | Save bookmark | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H3 | Search URL Area - Default Search Engine Promotion | Interacts with search url area | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H3 | Search URL Area - Homepage Setting Promotion | Interacts with search url area | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H3 | Open New Tab - New Tab Setting Promotion | Open new tab | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H4 | Open Bookmark - Pocket Promotion | Open bookmark | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H5 | Click on Pocket Stories - Pocket Promotion | Click on Pocket Stories | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
| H6 | Onboarding flow: Welcome slide -> FxAccount Sign up slide -> Search Settings banner -> Night Mode banner -> Bookmark banner -> Private browsing banner | First run | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
Android
- Android Hypothesis 1:
We believe that by prompting users to change their default browser to Firefox, we will increase DAUs and retention.
- Android Hypothesis 2:
We believe that by promoting add-ons to our users inside Firefox, we will increase DAUs and retention.
- Android Hypothesis 3:
We believe that by promoting Sync to our users inside Firefox, we will increase DAUs and retention as Firefox Account data shows that users with 2 synced devices retain longer.
- Android Hypothesis 4:
We believe that by promoting tips and tricks like the bookmark list or search engine settings to our users inside Firefox, we will increase DAUs and retention.
- Android Hypothesis 5:
We believe that by promoting Focus to our privacy conscious users inside Firefox, we will increase installs and DAUs in Focus, and increase retention on both apps (We know that iOS users who have Focus and Firefox installed retain better (Adjust)).
- Android Hypothesis 6:
We believe that introducing a new user to Firefox’s top features and get them using Firefox across devices will increase overall DAUs and retention of the Firefox browser.
- Android Hypothesis 7:
We believe by not promoting Pocket content to users who have opted out of Pocket in their Top Sites we are respecting the user choices and creating a better user experience.
- Android Hypothesis 8:
We believe by knowing when a user opens up the Firefox mobile app we can better measure the success of campaigns that encourage a user to open up the Firefox mobile app.
| Hyp. # | Message Name | Trigger | Targets |
|---|---|---|---|
| H1 | Launch app - Default Browser promotion | App Start | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H1 | Load Bookmark - Default Browser promotion | Load bookmark | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H2 | Launch app - Add-ons promotion | App start | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw in US, CA, AU, UK, DE, TW, HK, SG |
| H5 | Open New Tab - Focus promotion | Open new tab | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H4 | Save Bookmark - Bookmark List Promotion | Save bookmark | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H1 | Log in to FxA - Default Browser Promotion | Log in to FxA | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H1 | Add Bookmark 5 times - Default Browser Promotion | Add Bookmark 5 times | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H3 | Save Bookmark - Sync Promotion | Save bookmark | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H4 | Search URL Area - Default Search Engine Promotion | Interact with search url area | Test: en-US 10% of release population, Post-test: en, en-gb, en-ca, de, zh-tw inUS, CA, AU, UK, DE, TW, HK, SG |
| H6 | Onboarding flow: Original default onboarding slides -> Reader mode banner -> Bookmark banner -> Default browser banner | First run | Test: en-US 10% of release population, Post-test: en, en-gb, de, zh-tw |
For more detail about the triggers and deep links in Android, please go to this page[1]
