Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying that users are introduced to features that are unique and sticky

Ownership

• Developer contact:
• QA: Sorina Florean [:sorina]

Overall Status

[LANDED] Nightly 49: 2016-05-18

[LANDED] Aurora 49: 2016-06-08

[LANDED] Beta 49: 2016-08-03

Testing Summary

Scope of testing

The testing will be focus on:

• verifying that first-run is about features that are unique and sticky

Environments

Testing will be performed on following devices:

• Phones
  • 4.x
  • 5.x
  • 6.x
• Tablet
  • 4.x
  • 5.x
  • 6.x

Specifications

• Our belief is, if we can optimize the onboarding flow, specifically the first run experience, we will increase retention. In order to optimize the first run experience, we've ran a previous test (https://mozilla.aha.io/features/FENN-298) to validate the usage of multiple slides. The previous test was focused on testing a rather easy, "warm and fuzzy" story about Firefox and its core features. It's time now to validate which slides will perform the best and therefore we want to now compare this version with a stronger feature-focused slides where being a differentiatior to our competitors will be the key message. All feature are presented independently from each other and don't form a specific story arc (as opposed to the previous test).
  • Tab queue

• Reader View

• Sign up for Sync

User stories

• As a user, I want to be introduced to features that are unique and sticky for me to come back to the app and continue to use the browser as well as the features

Acceptance Criteria

• No discernible different in first-run browser start-up time

KPI/Metrics:

• Increased retention [requires A/B testing]
• Increased engagement

Testing details

Test Cases

• Test Case title
• https://docs.google.com/spreadsheets/d/1Ka39f93yd7fmSke9A6P9N3cW7p13elXhcrS5vDvesX0/edit#gid=0

Testing days

Date: 2016-06-12

• Created Test Plan

Date: 2016-06-16

• Exploratory on latest Aurora with Motorola Razr (Android 4.4.4), LG G4 (Android 5.1) and Nexus 5 (Android 6.0.1)
  • Tab queue Aurora first-run

• Reader View Aurora first-run

• Sign up for Sync Aurora first-run

• Verified bugs:
  • bug 1268647 - Implement new set of feature-focused firstrun panels
  • bug 1276374 - Remove firstrun Content Notifications slide

Date: 2016-07-12

• Exploratory testing on latest Aurora and Nightly with:
  • LG G4 (Android 5.1)
  • Nexus 5 (Android 6.0.1)
  • Prestigio Grace X5 (Android 4.4.2)
  • Asus Transformer Pad (Android 4.2.1)
  • Asus ZenPad 8 (Android 5.0.2)

Date: 2016-08-08

• Exploratory testing on Firefox 49 Beta 1 with LG G4 (Android 5.1) and Nexus 5 (Android 6.0.1)
  • Tab queue - first-run

• Reader View - first-run

• Sign up for Sync - first-run

Date: 2016-09-19

• Exploratory testing on Firefox 49 RC with Motorola Razr (Android 4.4.4), Asus ZenPad 8 (Android 5.1) and Nexus 5 (Android 6.0.1)

Bug Work

• Meta bug
  • bug 1268641 - (feature-onboarding) Feature-based onboarding

Full Query

ID	Priority	Summary	Status	Assigned to
1268647	--	Implement new set of feature-focused firstrun panels	VERIFIED	Chenxia Liu [:liuche]
1270993	--	Confirm that we have a sync probe for "have sync account"	RESOLVED
1276374	--	Remove firstrun Content Notifications slide	RESOLVED	Chenxia Liu [:liuche]
1282968	--	Redash android_events_v1.experiments missing onboarding3-a	RESOLVED	Chenxia Liu [:liuche]

4 Total; 0 Open (0%); 3 Resolved (75%); 1 Verified (25%);

Still issues

Bug fixed

[RESOLVED FIXED] Bug 1268647 - Implement new set of feature-focused firstrun panels
[RESOLVED FIXED] Bug 1276374 - Remove firstrun Content Notifications slide
[RESOLVED FIXED] Bug 1270993 - Confirm that we have a sync probe for "have sync account"

Signoff

Criteria

• All test cases should be executed
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed

Results

Nightly testing

• There were no blockers found while testing this feature.
• All bugs are fixed and verified

Aurora testing

• There were no blockers found while testing this feature.
• All bugs are fixed and verified

Beta testing

• There were no blockers found while testing this feature.
• All bugs are fixed and verified