QA/Screenshots54
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
07/21/2016 | 1.0 | Vlad Bacia | Created first draft |
02/28/2017 | 1.1 | Cosmin Muntean | General Updates |
03/06/2017 | 1.2 | Peter deHaan | Added new FF 54 Plans |
04/25/2017 | 1.3 | Cosmin Muntean & Paul Oiegas | Updates for Screenshots system add-on |
Contents
Overview
Purpose
Firefox Screenshots (Project Name: Page Shot) is an in-browser screenshoting application that lets you save and share permanent copies of anything on the web. It was originally deployed through Test Pilot as an experiment.
Scope
This wiki details the testing that will be performed by the project team for the Screenshots project. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
Engineering:
- Engineering - Ian Bicking
- Engineering - Donovan Preston
- Engineering - Jared "6a68" Hirsch
- Engineering - Danny Coates
- UX - Bram Pitoyo
- Ops - Daniel "relud" Thorn (backup is Miles)
QA:
- PM for QA team - Rares Bologa
- Leading QA efforts - Paul Oiegas
- QA - Cosmin Muntean
- QA - Peter deHaan
Management:
- Product Manager - John Gruen
- Sr. Engineering Manager - Wil Clouser
- Engineering Program Manager - Cory Price
Communication channels:
IRC - #screenshots
Contact email - screenshots-staff@mozilla.com (goes to Screenshots staff)
Testing summary
Currently there are no automated tests for Screenshots, apart from a load_test_script and Selenium smoke test, although standard8 is going to set up a MochiTest for Screenshots (See #2320).
Scope of Testing
In Scope
Screenshots lets you save anything on the web, including pages that requires a login or content that changes or is personalized. Once you've created a shot, anyone you give the link to can view it; never worry about logins or disappearing content. Screenshots is built and hosted by Mozilla, so you know it's safe to use.
The scope of our testing is the Screenshots-desktop system add-on and its functionality. The testing effort for Screenshots will be invested on the following areas:
- integration: verify the system add-on integration with the current browser functionalities and UI;
- functionality: basic and advanced functionality to be verified according to the existing requirements;
- usability: Screenshots features must be easy to use and straightforward;
- load testing: Verify that the server can handle expected load; pageshot-loadtests
Out of Scope
The mobile implementation and testing.
Requirements for testing
Environments
Testing will be performed on the following OSes (both x86 & x64 infrastructures):
- Windows 7
- Windows 8.1
- Windows 10
- Ubuntu 14.04
- Ubuntu 16.04
- Mac OS X 10.10
- Mac OS X 10.11
- Mac OS X 10.12
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
Ref | Function | Test Objective | Evaluation Criteria | Test Type | Owners |
---|---|---|---|---|---|
1 | Screenshots Firefox button | Verify that the Screenshots button is displayed in the Firefox browser toolbar. | Screenshots scissor button is displayed and active in the toolbar. | Manual | Eng Team |
2 | Screenshots Onboarding tour | Verify that when the Screenshots button is clicked, the Onboarding tour is displayed. | Tour is displayed and describes the Screenshots functionality. | Manual | Eng Team |
3 | Screenshots functionality | Verify all Screenshots functionalities are working. | 1. Save anything on the web. 2. Share content. |
Manual | Eng Team |
3 | Disable / Re-enable Screenshots system add-on | Verify that the system add-on can be disabled and enabled without any issues. | 1. Disable Screenshots system add-on from "about:preferences"page. Or from "about:config"page, using extensions.screenshots.system-disabled pref ("True" for disabled and "False" for enabled). 2. Verify that the browser UI returned to default functionality on the affected areas. 3. Re-enable Screenshots system add-on and verify that the browser has changed back. |
Manual | Eng Team |
Builds
Firefox Screenshots system add-on can be found in the next builds:
- Nightly - starting with build from 2017-04-18
- Beta - starting with Beta 54.0b2 2017-04-25 (candidate)
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 8/02/2015 | - |
Study documentation/specs received from developers | 07/19/2016 | 07/20/2016 |
QA - Test plan creation | 07/21/2016 | 07/22/2016 |
QA - Test cases/Env preparation | 09/01/2016 | 09/09/2016 |
QA - Beta Testing | 04/25/2017 | |
Load Testing | 03/01/2017 | 03/30/2017 |
Release Date | 06/13/2017 |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail |
Test case execution | TestRail |
Bugs management | GitHub |
Status
Overview
- Nightly - starting with build from [2017-04-18]
- Beta - starting with Beta 54.0b2 [2017-04-25]
Risk analysis
Risk areas | Mitigation |
---|---|
External dependencies | - We should do an end-to-end test of how we will integrate with the browser and the server. - We should determine what our exact requirements are for sample size. - Look into Telemetry as a contingency plan. |
Screenshots system add-on compatibility and usability with content specific websites | - Pictures and Videos from embedded sources. - Private websites and private content. |
Scope | - All locales - Accessibility requirements == https://github.com/mozilla-services/screenshots/issues/2251 |
Tests | - Lack of comprehensive server unit tests (https://github.com/mozilla-services/screenshots/issues/427) |
Open issues | - Number of open issues not declining (https://github.com/mozilla-services/screenshots/issues), although there aren't currently any beta/release blockers identified. |
References
List and links for specs:
- Screenshots Wiki page.
- Screenshots Dev website.
- Screenshots Stage website.
- Screenshots Production website.
- Screenshots GitHub repository.
- Screenshots Privacy Statement.
- Screenshots Metrics.
- Screenshots go-live metrics plan.
- Screenshots UX and Measurements Acceptance Criteria.
- Testing the API
Meta bug:
- [meta Shipping Firefox Screenshots]
Testcases
Overview
Summary of testing scenarios
Test Areas
Test Areas | Covered | Details |
---|---|---|
General | ||
Private Window | Yes | |
Multi-Process Enabled | Yes | |
Multi-process Disabled | Yes | |
Theme (high contrast) | Yes | |
Enforce PNG check when posting images | Yes | See #2301 |
L20n | ||
Verify Screenshots is available in core set of supported locales | Yes | See: #2295, #2294, and #2286 |
UI | ||
Mouse-only operation | Yes | |
Keyboard-only operation | No | See #2251 |
Display (HiDPI) | Yes | |
Interaction (scroll, zoom) | Yes | |
Usable with a screen reader | No | e.g. with NVDA |
Usability and/or discoverability testing | Yes | Is this feature user friendly |
Install/Upgrade/Uninstall | ||
Feature upgrades/downgrades data as expected | Yes | |
Does sync work across upgrades | No | |
Requires install testing | No | |
Verify user data is migrated from Test Pilot experiment | Yes | See #2260 |
Affects first-run or onboarding | No | |
Does this affect partner builds? Partner build testing | No | We won't have partner builds available for testing in the Nightly or Aurora channels |
Verify that Screenshots can be "disabled" | Yes | It should be able to be enabled and disabled by a toggle in about:preferences or by pref in about:config |
Data Monitoring | ||
Temporary or permanent telemetry monitoring | - | Not covered by the Engineering QA team |
Telemetry correctness testing | - | Not covered by the Engineering QA team |
Verify pings are sent when clicking Save addon/save-shot/overlay-save-button |
Yes | |
Verify pings are sent when clicking Cancel addon/cancel-shot/overlay-cancel-button |
Yes | |
Verify pings are sent when clicking Download; addon/download-shot/overlay-download-button |
Yes | |
Verify pings are sent when clicking My Shots; addon/goto-shots/selection-button |
Yes | |
Verify pings are sent when clicking Save Visible; addon/capture-visible/selection-button |
Yes | |
Verify pings are sent when clicking Save Full Page; addon/capture-full-page/selection-button |
Yes | |
Record if Screenshots is enabled or not via Telemetry | Yes | See #2304 |
Server integration testing | - | Not covered by the Engineering QA team |
Offline and server failure testing | - | Not covered by the Engineering QA team |
Load testing | Yes | See screenshots-loadtests page |
Add-ons | ||
If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | ||
Addon API required? | No | |
Comprehensive API testing | No | |
Permissions | No | |
Testing with existing/popular add-ons | Yes | |
Security | ||
Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | ||
3rd-party security review | - | Not covered by the Engineering QA team |
Privilege escalation testing | - | Not covered by the Engineering QA team |
Fuzzing | - | Not covered by the Engineering QA team |
Web Compatibility | ||
Testing against target sites | Yes | |
Survey of many sites for compatibility | No | |
Interoperability | ||
Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | ||
Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | No | |
Interaction of this feature with other browser features | Yes |
Test suite
- Full Test suite - Screenshots Full Functional (TestRail)
- Smoke Test suite - TBD if available/needed
- Regression Test suite - TBD if available/needed
Bug Work
- Tracking bug – GitHub issues page
- Found issues list - SV QA Engineering found bugs
Sign off
Criteria
Check list
- Should meet the UX and Measurements Acceptance Criteria
- All test cases should be executed
-
Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan - All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite - Full Functional run from 04/18/2017
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run - N/A
- Full Test suite - N/A
Beta Testing
List of OSes covered by testing: Mac OS X 10.11.6, Ubuntu 16.04 x64, Windows 10 x64, Windows 7 x64
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | [DONE] | |
Testing Infrastructure setup | No | |
Test Plan Creation | [DONE] | |
Test Cases Creation | [DONE] | |
Full Functional Tests Execution | [DONE] | |
Automation Coverage | - | Not on QA manual end. |
Performance Testing | - | Not on QA manual end. |
All Defects Logged | [IN PROGRESS] | |
Critical/Blockers Fixed and Verified | [IN PROGRESS] | |
Daily Status Report (email/etherpad statuses/ gdoc with results) | Status etherpad | |
Metrics/Telemetry | ||
Load testing | [IN PROGRESS] | |
QA Beta - Full Testing | [DONE] | |
QA Signoff - Beta Release | [DONE] | (YELLOW) Pre-Release Report sent on 07/21/2017 |