Add-ons/QA/Testplan/Auto-Approvals
Revision History
| Date | Version | Author | Description |
|---|---|---|---|
| 04/21/2017 | 1.0 | Valentina Virlics | Created first draft |
Contents
Overview
Currently, every add-on version listed on AMO is manually code reviewed by a human (called: reviewer). This process is meant to ensure a good user experience with add-ons. This manual review process requires significant human effort and this is why we need auto-approvals - a change that will be implemented in the review process, Reviewer Tools and submission to move to a hybrid system that includes human reviews as well as automatic reviews.
Purpose
Reducing delays in reviews (delays present due to decreased reviewer activity and increase submissions) by automatically reviewing some add-ons (that meet a list of criteria).
Entry Criteria
Current Status
- The feature was released.
Exit Criteria
- All related bugs triaged
- All blockers fixed
- All resolved bugs verified by QA
- Found-fixed bugs rate going down in time
Scope
what's in scope?
- Add a counter for approvals since last automatic review
- Get extensions auto-approved if they meet the following criteria:
- Have been manually approved once
- Are not flagged for admin review (this can happen automatically on submission in some cases, but is generally done manually by reviewers)
- Are WebExtensions
- Active Daily Users are under 10K (initially set to this value, but configurable).
- Are not triggering any of the following linter warnings:
- CSP
- Native Messaging
Not yet implemented:
- Auto-approve extensions that are not triggering content scripts warnings for all URLs at validation
what's out of scope?
- Human unreviewed extensions
- Embedded Webextensions
Ownership
Product Manager: Jorge Villalobos; irc nick :jorgev
QA Manager: Krupa Raj; irc nick :krupa
QA Lead: Victor Carciu; irc nick :victorc
Add-ons QA: Valentina Virlics; irc nick :ValentinaV
Requirements for testing
Environments
- Windows
- Mac OS
Servers
- Stage: https://addons.allizom.org/en-US/
- Dev: https://addons-dev.allizom.org/en-US/
- Production: https://addons.mozilla.org/en-US/
Channels
- Release
Test Strategy
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 31.01.2017 | |
| Study PRD/mocks received | 01.02.2017 | |
| QA - Test plan creation | 21.04.2017 | |
| QA - Test cases preparation | 21.04.2017 | |
| QA - Test cases execution | 24.04.2017 | |
| Release Date | 27.04.2017 (Q1 - 2017) |
Created an etherpad with more detailed scenarios - Auto-Approvals
Testing Tools
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail |
| Test case execution | TestRail |
| Bugs management | Github |
References
* PRD - https://docs.google.com/document/d/1VxvvzQQvfDOVCylSVlpfeXv0tDYRLZfIMvWOoraANGU/edit# * Tracker: 4533
Testcases
Test Areas
- Submission Tools/Reviewer Tools
Test suite
Full Test Suite
Bug Work
- Feature implementation bug - 4533
Open bugs
- 325 - Automatically flag add-ons using native messaging for admin review
- 5273 - Add ability to store reviewer approval for auto-approved add-ons
- 5274 - Stop sending email to developers when requesting super-review of an auto-approved add-on
- 5275 - Implement rejection of auto-approved add-ons by human reviewers
- 5292 - Add locked/has_info_request/has_admin_review flags to AutoApprovalSummary
Closed bugs
- 5155 - Automate auto-approval task
- 5154 - Add a list of auto-approved add-ons in reviewer tools
- 4752 - Maintain a counter of the number of times an add-on has been approved
- 4855 - Set up auto_approve command logging to be shown in syslog and console
- 4730 - Remove old auto validation code
- 5164 - Reset AddonApprovalsCounter for add-ons when they are auto approved
Sign off
Criteria
Check list
- All test cases should be executed
- All blockers must be fixed and verified or have an agreed-upon timeline for being fixed
