Overview

Purpose

The purpose of this wiki is to serve as a general test plan for verifying Desktop Search engine works as expected after the refactoring.

Scope

The testing will be focus on

• verify that search works correct
• verify that customized user data/profile is not affected after update

Testing will be performed manually, by QA engineering team.

Ownership

Developer contacts: Florian Queze
QA:

Brindusa Tot - leading QA efforts

Simona Badau, QA

Ovidiu Boca, QA

Testing Summary

Scope of testing

In Scope

The main testing scope is to assure that the search works correct. For this fix, testing will include steps where users updates profile and we should verify that the user profile is not affected.

Requirements for testing

Environments

Testing will be performed on following OSes (both x86 & x64 infrastructures):

• Mac OS X 10
• Windows 7
• Windows 8
• Windows 8.1
• Windows 10
• Windows XP
• Windows Vista
• Ubuntu 14.04

Testing on specific devices is not needed.

Test Strategy

Test Objectives

Following table contains main features to be tested:

Ref	Function	Test Objective	Evaluation Criteria
1	Search - basic scenarios	Search using different settings and from different broswer locations	Search is performed correctly.
2	Search - user profile verification after update	Validate that after update to latest firefox user profile is not affected	User profile is not updated
3	Search - hijack search with external apps	Try to hijack firefox search engine	Validate that search engine are not hijacked

Test Types

Type of Testing	Required (Yes/No)	Owners	Details/Comments
Manual	Yes	Moz QA eng team
Automation	Yes	Florian Queze
Performance	No
Security	No
Usability	No
Metrics/Telemetry	TBD

Builds

This section should contain links for builds with the feature -

Nightly builds with the search refactor fix can be reached at link

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase	Start Date	End Date
Start project	10.20.2015
Study documentation/specs received from developers	10.20.2015	-
QA - Test plan creation	10.23.2015
QA - Test cases/Env preparation	10.23.2015	10.30.2015
QA - Nightly Testing	10.30.2015	11.25.2015
QA - Aurora Testing	11.30.2015	01.25.2016
QA - Beta Testing	01.26.2016
Release Date

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process	Tool
Test plan creation	Mozilla wiki
Test case creation	Google docs
Test case execution	Google docs
Bugs management	Bugzilla

Status

Overview

[LANDED] Nightly 45: first landed on - 10.29.2015

Risk analysis

- No Risks found at the moment

References

Meta: Bug 1203167

Test cases

Overview

• Perform search from the Search Bar – using different terms
• Perform search from all the available search fields
• Add/Update search engine and validate search results is given by the new engine
• Change search settings (select/deselect engine, remove/restore engine, reorder engines) and validate search results is given by the new engine
• Verify if custom search engine settings (new engines added, reordered engines) are preserved after update
• Verify that after update, search engines settings can be updates
• Install external software's that automatically change the search engine for installed browsers and verify that the default search engine or users custom search engine is not altered

Test Areas

Test Areas	Covered	Details
Private Window	[DONE]
Multi-Process Enabled	[DONE]
Multi-process Disabled	[DONE]
Display (HiDPI)	N/A
Interraction (scroll, zoom)	N/A
Theme (high contrast)	N/A

Test suites

Full Test Suite - can be reached at link

Bugs Work

Tracking bug: Meta bug 1203167

Sign off

Criteria

• All test cases should be executed
• Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
• All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Nightly testing

• [DONE] Full Test suite results [1]
  
  • There were no blockers found while testing this feature.
  • Manual testing covered Windows 7 x64, Windows 8 x64, Windows 8.1 x32 Windows 10 x64, Windows XP x64, Windows Vista x32, Ubuntu 14.04 x32, Ubuntu 15.04 x 64, Mac OS x 10.10, Mac OS x 10.11.

[DONE] Merge to Aurora Sign-off (2015-12-14)

Mid-Aurora Sign-off (2016-01-11)

• Full Test suite results [2]
  
  • Manual testing covered Windows 7 x64, Ubuntu 14.04 x32 and Mac OS X 10.9.5.
  • There were no new issues found during sign-off.

Pre-Beta Sign-off (2016-01-22)

• • Smoke testing covered Windows 7 x64, Ubuntu 14.04 x64 and Mac OS X 10.9.5.

Pre-Release Sign-off (2016-02-29)

• • Smoke testing covered Windows 7 x64, Ubuntu 14.04 x64 and Mac OS X 10.9.5.

Checklist

Exit Criteria	Status	Notes/Details
Testing Prerequisites (specs, use cases)	[DONE]	Bug 1203167
Testing Infrastructure setup	[DONE]
Test Plan Creation	[DONE]
Test Cases Creation	[DONE]
Full Functional Tests Execution	[DONE]
Performance Testing	N/A
All Defects Logged	[DONE]
Critical/Blockers Fixed and Verified	[DONE]
Metrics/Telemetry	N/A
UAT Test Cases (signoff By Users)	N/A
QA Signoff - Nightly Release	[DONE]	Email sent: Search SignOff email
QA Postmortem	[DONE]	Meeting Notes