QA/Mobile/FocusAndroidTestPlan

From MozillaWiki
< QA
Jump to: navigation, search

This is the Test Plan for Focus for Android (https://github.com/mozilla-mobile/focus-android)

Intro/Summary/Notes

Intro and Summary

  • This wiki outlines the test requirements for Focus for Android releases.
  • Use this generalized Test Plan as a starting point for creating Plans, Suites, and Cases

Notes

Feature Lists

For the list of features and its status, please refer to here. 'Aqua' label refers to the issues that are slated for the next upcoming release

  • NOTE: Focus for Android bugs are maintained in Github

Schedule

  • All features for 2.0 should be completed by Aug 4th, 2017 to begin acceptance testing
  • Testing, including acceptance testing, should be completed by Aug 10th, 2017.
  • Focus for Android 2.0 is currently scheduled to be released Aug 17th, 2017.

Personnel

QA Team

Resource Plan

  • Each QA person will be 40% allocated to this project

Test Strategy

Process

Current Sprint

SUMMARY

  1. Label Issues
  2. Create/Update Test Cases
  3. Verify Issues

DESCRIPTION

  • When the development work on github issue is completed, whether before/after code freeze, the issue will be closed so that the issue is available for testing. If a bug requires extra attention, it'll be marked with 'QAReady' label.
  • The issue should belong to a correct milestone before development work can be started and may include either new features or bugs.
  • Once an issue has been closed, when applicable, a (new) test case should be added to /updated in TestRail.
  • TestRail type field should be set to: "Functional" and automation field to: "Untriaged"
  • QA will verify the issue, and make sure to outline clear test steps in the (new) TestRail test case.
  • Upon completion, github issue should be marked with QAVerified, removing QAReady label if exists.
  • After the QA signoff, ideally, all verifiable issues should be marked with QAVerified.
  • NOTE: the meta-label testing is also added by both devs and QA for any bug or feature requiring testing of any kind).
  • Once the issue is closed, if suitable for automation, label the it in github as automation-TBD and change the automation dropdown in TestRail to "Suitable".

Current or Post Sprint

  • Query github for issues labelled "automation-TBD"
  • Create new automated test(s) to verify proper function of feature based on steps outlined in TestRail.
  • Cross-link TestRail test case within automated test (add link to java test file on github in Pre-Conditions section)
  • Cross-link automated test to TestRail test case (add link to TestRail test case in java file).
  • Submit PR for review and, once UI tests are passing, change github label to automated and set TestRail type to: "Automated" and automation field to: "Completed"

QA Criteria

  • Once Dev team announces code freeze, a date for the QA signoff is decided, usually a week from the code freeze.
  • QA will run the regression test suite, updated with the coverage for new features, and inform the Dev team with any findings.
  • During triage, it will be decided that whether a found issue is a release blocker or not. As a minimum, a bug shall be a release blocker if it causes crash/hang on a relatively common use case scenario. The Dev team will provide fix for the release blockers during the code freeze, and the fix will be verified.
  • QA will sign off the release when the (up-to-date) regression test suite run has been completed, all items under Test_Deliverables has been completed, and there are no release blockers.

Manual Tests

UI test case suite for project: "Focus for Android" will be located in TestRail. As a minimum, each of the P1 UI features will have corresponding test cases. The test suite will be executed by Softvision for the acceptance testing.

Automated Tests

Automation tests will be mainly used for quick regression check of key UI features. Overall, automation tests will be added to the same Github repo, and executed on Google Firebase. While the unit tests will be run for each commit, UI Tests (in Espresso/UIAutomator framework) will be executed in master branch only. The test result will be inspected for new failures, and github issue will be raised for tracking.

Automation tests are located here.

In addition to regularly run automated tests, QA will also generate screenshots for l10n verification.

Code Analysis

  • Codecov is added to the github repo, and will display increase/decrease of unit test code coverage
  • Findbugs addon is added to warn developers against possible code issues, as well as Lint.

Outside Scope

Following tests are outside the scope of the QA:

  • Unit Tests
  • Performance Tests
  • Localization Tests
  • Beta testing with a wider audience

Test Deliverables

  • Completed TestRail Test suite which covers all features with UI aspects
  • Automated Test suite in master branch, running on every checkin
  • Github issues for every issue found
  • Manual acceptance test report
  • Signoff Decision

Testing Days

June 14th 2017:

July 14th, 2017 (Ioana Chiorean):

July 14th, 2017 (Oana Horvath):

August 24th/25th, 2017 (Ioana Chiorean):

September 9th, 2017 (Ioana Chiorean):

December 6th, 2017 (Sorina Florean):

Dependencies/Risks

  • Since the blocking status of trackers is not exposed via adb log or UI, its validation will rely on unit tests
  • The performance of Focus for Android is not measured precisely, will rely on the tester feedback
  • The tester pool for a preliminary first release will be extremely small
  • The automation/manual test suites will be built incrementally as new features are added to Github, and the late features may not be exposed to validation as much as earlier ones. The acceptance test after the feature completion date will be performed with the special emphasis on 1. key features and 2. most recently added features

Signoff/Exit Criteria

  • No critical or major UI bugs
  • No easily reproducible crashes
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Mobile/FocusAndroidTestPlan&oldid=1205458"