QA/Captive Portals
From MozillaWiki
< QA
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
| Date | Version | Author | Description |
|---|---|---|---|
| 10/27/2016 | 1.0 | Adrian Florinescu | Initial Draft |
| 11/08/2016 | 1.1 | Adrian Florinescu | TP review with Nihanth; updates based on it. |
| 11/21/2016 | 1.2 | Adrian Florinescu | Updates |
| 02/01/2017 | 2.0 | Adrian Florinescu | Updates: preparing for hand-over to RelQA |
Contents
Overview
Captive Portal feature covers the detection and implementation of handles for captive portals inside Firefox browser. Firefox is expected to handle the handling of a captive portal page upon detection of such. Certain business logic related to the detection and management of a captive portal is expected to be in place and documented in the Specs section.
Purpose
Detail the purpose of this document. For example:
- The test scope, focus areas and objectives
- The test responsibilities
- The test strategy for the levels and types of test for this release
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Scope
This wiki details the testing that will be performed by the project team for the Captive Portals feature. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
• Program Management:
• Product Management:
• Engineering Management:
• Eng.:
• QA
Testing summary
Scope of Testing
In Scope
- testing a wifi-connection that has enabled captive portals
- testing a wan connection which has enabled captive portals (not done)
Out of Scope
- Due to security issues and since the types of captive portals implementation vary depending on manufacturer, setup, security setups, etc., the tests which will be designed and ran will cover the basic functionality that impacts Firefox browser and not the compatibility with all existing captive portals.
Requirements for testing
Environments
- standard OS's: Ubuntu, Windows, OSX (versions TBD)
- Captive Portal compliant and activated - wifi network
- Wireless connection available
Channel dependent settings (configs) and environment setups
Nightly
Captive Portal feature active by default.
Beta
Captive Portal feature active by default.
Post Beta / Release
Not Active.
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered. Please note that this is at a high level. For large projects, a suite of test cases would be created which would reference directly back to this master. This could be documented in bullet form or in a table similar to the one below.
| Ref | Function | Test Objective | Test Type | Risk Assessment and Coverage(RAC) | Owners |
|---|---|---|---|---|---|
| TO-1 | Active Portal detection | To verify that the active portal is detected in all expected scenarios | Manual | RAC-1, RAC-2, RAC-3, RAC-4,RAC-8 | Eng Team |
| TO-2 | Active Portal notification bar | To verify the behavior of the active portal notification bar | Manual | RAC-5 | Eng Team |
| TO-3 | Active Portal Tab | To verify that new tab is opened and it's further behavior according to specs on active detection | Manual | RAC-6, RAC-7 | Eng Team |
| TO-4 | Inactive Portal | To verify the behavior of FF when the captive portal is not active anymore | Manual | - | Eng Team |
| TO-5 | Preferences | To verify the preference - on/off | Manual | - | Eng Team |
| TO-6 | Error page | To verify the Error page | Manual | - | Eng Team |
| TO-7 | Localization | To verify that the Notification bar, Error page and any other element is localized | Manual | - | Eng Team |
Risk Assessment and Coverage
| ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
|---|---|---|---|---|---|---|
| RAC-1 | Multiple FF windows | TO-1 | 2-Moderate | 3-High | 2-Medium | 12 |
| RAC-2 | Multiple network connections (wi-fi x2/ wi-fi - wan) | TO-1 | 1- Low | 1-Unlikely | 2-Medium | 2 |
| RAC-3 | Existing proxy settings | TO-1 | 2-Moderate | 2-Possible | 2-Medium | 8 |
| RAC-4 | Integration with (Windows) Captive Portal managers | TO-1 | 2-Moderate | 3-Almost Certain | 2-Medium | 12 |
| RAC-5 | Notification management against Pop-up blockers | TO-2 | 3-High | 2-Possible | 3-High | 18 |
| RAC-6 | (OSX) Multi users | TO-3 | 2-Moderate | 2-Possible | 2-Medium | 8 |
| RAC-7 | (OSX) Integration with Captive Network Assistant | TO-3 | 2-Moderate | 2-Possible | 3-High | 12 |
| RAC-8 | (Ubuntu)Circumventing Captive portal | TO-1 | 2-Moderate | 2-Possible | 3-High | 12 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Builds
This section should contain links for builds with the feature -
- Links for Nightly builds
- Links for Aurora builds
- Links for Beta builds
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
| Project phase | Start Date | End Date |
|---|---|---|
| Start project | 24.10.2016 | |
| Study documentation/specs received from developers | 10.31.2016 | 11.08.2016 |
| QA - Test plan creation | 10.31.2016 | 11.08.2016 |
| QA - Test cases/Env preparation | 10.31.2016 | |
| QA - Nightly Testing | ||
| QA - Aurora Testing | ||
| QA - Beta Testing | ||
| Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
| Process | Tool |
|---|---|
| Test plan creation | Mozilla wiki |
| Test case creation | TestRail/ Google docs |
| Test case execution | TestRail |
| Bugs management | Bugzilla |
Status
Overview
Track the dates and build number where feature was released to Nightly Track the dates and build number where feature was merged to Aurora Track the dates and build number where feature was merged to Release/Beta
References
- List and links for specs
https://public.etherpad-mozilla.org/p/captive-portal-functionality https://wiki.mozilla.org/Necko/CaptivePortal
- bug 1202680 - [userstory] Automatic Captive Portal detection
37 Total; 9 Open (24.32%); 20 Resolved (54.05%); 8 Verified (21.62%);
Testcases
Test Areas
| Test Areas | Covered | Details (and why) | Reviewed by |
|---|---|---|---|
| Private Window | Yes | Captive portal detection should work in Private window as well | Nihanth Subramanya |
| Multi-Process Enabled | Yes | By default | Nihanth Subramanya |
| Multi-process Disabled | Yes | The Captive Portal detection should be verified on non-e10s | Nihanth Subramanya |
| Theme (high contrast) | Yes | Because of the notification bar and error page | Nihanth Subramanya |
| UI | |||
| Mouse-only operation | Yes | Captive Portal feature is entirely mouse only | Nihanth Subramanya |
| Keyboard-only operation | Yes | The Show login button should be accessible by keyboard only | Nihanth Subramanya |
| Display (HiDPI) | Yes | Because Notification bar and Error page | Nihanth Subramanya |
| Interraction (scroll, zoom) | Yes | Because of Error page and Notification bar | Nihanth Subramanya |
| Usable with a screen reader | Yes | The notification bar/Show Login button should be recognizable by screen reader | Nihanth Subramanya |
| Usability and/or discoverability testing | Yes | Is this feature user friendly | Nihanth Subramanya |
| RTL build testing | Yes | Because of Error page and Notification bar | Nihanth Subramanya |
| Help/Support | |||
| Help/support interface required | TBD | Learn more still not implemented. | Nihanth Subramanya |
| Support documents planned(written) | TBD | Learn more still not implemented. | Nihanth Subramanya |
| Install/Upgrade | |||
| Feature upgrades/downgrades data as expected | No | feature doesn't upgrade/downgrade | Nihanth Subramanya |
| Does sync work across upgrades | No | Not applicable for this feature | Nihanth Subramanya |
| Requires install testing | No | No separate feature/application installation needed (not only Firefox) | Nihanth Subramanya |
| Affects first-run or onboarding | Yes | To check first run, since there might be problems having the tour interfere with captive portal | Nihanth Subramanya |
| Does this affect partner builds? Partner build testing | No | It's a core functionality, it's expected not to matter if the build is partner or not | Nihanth Subramanya |
| Enterprise | Raise up the topic to developers to see if they are expecting to work different on ESR builds | Nihanth Subramanya | |
| Enterprise administration | No | For the moment, this isn't in scope. | Nihanth Subramanya |
| Network proxies/autoconfig | No | For the moment, this isn't in scope. | Nihanth Subramanya |
| ESR behavior changes | No | For the moment, this isn't in scope. | Nihanth Subramanya |
| Locked preferences | No | For the moment, this isn't in scope. | Nihanth Subramanya |
| Data Monitoring | |||
| Temporary or permanent telemetry monitoring | Yes | There is plan, but it's not designed to be MVP'ed for 52, to reevaluate further on | Nihanth Subramanya |
| Telemetry correctness testing | No | See above point. | Nihanth Subramanya |
| Server integration testing | No | Not in scope. | Nihanth Subramanya |
| Offline and server failure testing | No | Not in scope. | Nihanth Subramanya |
| Load testing | No | Not in scope. | Nihanth Subramanya |
| Add-ons | If add-ons are available for testing feature, or is current feature will affect some add-ons, then API testing should be done for the add-on. | ||
| Addon API required? | No | Not in scope | Nihanth Subramanya |
| Comprehensive API testing | No | Not in scope | Nihanth Subramanya |
| Permissions | No | Not in scope | Nihanth Subramanya |
| Testing with existing/popular addons | Yes | Because the Notification bar might be blocked by Pop-up blockers | Nihanth Subramanya |
| Security | Security is in charge of Matt Wobensmith. We should contact his team to see if security testing is necessary for current feature. | ||
| 3rd-party security review | TBD | (how we find out if it needs to have 3rd partyt sec review)? | Nihanth Subramanya |
| Privilege escalation testing | No | Nihanth Subramanya | |
| Fuzzing | No | Nihanth Subramanya | |
| Web Compatibility | depends on the feature | ||
| Testing against target sites | No | Not significant for this feature. | Nihanth Subramanya |
| Survey of many sites for compatibility | No | Not significant for this feature. | Nihanth Subramanya |
| Interoperability | depends on the feature | ||
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers. | Yes | Yes, because there are OS Captive portal managers for FF Captive Portal detection to be tested against. | Nihanth Subramanya |
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS | TBD | AT the moment, there is no plan to have this feature implemented on anything else than Desktop | Nihanth Subramanya |
| Interaction of this feature with other browser features | No | There are no significant reasons to believe that Captive Portal might influence other FF areas. | Nihanth Subramanya |
Test suite
Full Test suite - Test Rail
Bug Work
Tracking bug – meta bug Bug 1202680 Automatic Captive Portal detection
NEW bugs
| Bug No | Summary | Status | Firefox Version |
|---|---|---|---|
| 1280599 | Handle captive portal page not loading | NEW | |
| 1303775 | Captive portal tab may stay open after login | NEW | 54:affected; 53:fix-optional; 52:wontfix |
| 1313577 | Forget captive portal tab after origin change | NEW | 54:affected; 53:fix-optional; 52:wontfix |
| Bug 1315966 | CaptivePortalDetector should propagate login page URL when it's available | NEW | 52:fix-optional |
| Bug 1315967 | CaptivePortalService should provide the URL of the login page if it's available | NEW | 54:affected; 53:fix-optional; 52:fix-optional |
| Bug 1320327 | [Captive Portal] Captive Portal notification/Login are shown even though the Captive portal canonical link is pre-loaded | NEW | 53:affected; 52:wontfix |
| Bug 1320330 | [Captive Portal] With Voice Over on, text from Captive Portal notification bar is not read | NEW | 53:affected |
| Bug 1332630 | [Captive Portal] [Accessibility] Keyboard only doesn't focus the Captive Portal notification bar | NEW | 53:affected; 52:affected |
| Bug 1315968 | Investigate how to include information about captive portal state in telemetry data/error reports | NEW | 54:affected; 53:fix-optional; 52:fix-optional |
VERIFIED ON 53, NEED UPLIFT ON 52
Bug 1318640 - Closing the captive portal tab should go back to the previously selected tabs
Bug 1322201 - Ensure correct usage of "login" vs "log in" in captive portal strings
Resolved WONTFIX
Bug 1323514 - [Captive portal] Banner shown on tabs when visiting local network sites
Bug 989195 - Replace about:newtab and about:home with captive portal if detected
Resolved INVALID
Bug 1329546 - Get rid of CaptivePortalWatcher and unify captive portal tests
Bug 1320113 - [Captive Portal] Notification bar and Show login button translation tracker
Bug 1320100 - [Captive Portal] In full screen the notification bar doesn't appear
Resolved WORKSFORME
Bug 989196 - Do not show open Captive Portal automatically when OSX Captive Network Portal Assistant has been opened
Bug 989192 - Create portal detection events on network topology changes and user login
Sign off
Criteria
Check list
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Merge to Aurora Sign-off
Checklist
| Exit Criteria | Status | Notes/Details |
|---|---|---|
| Testing Prerequisites (specs, use cases) | No | There is available documentation, but currently it's all centralized into a Etherpad. |
| Testing Infrastructure setup | Yes | |
| Test Plan Creation | Yes | |
| Test Cases Creation | Yes | |
| Full Functional Tests Execution | Yes | |
| Automation Coverage | TBD | |
| Performance Testing | No | |
| All Defects Logged | Yes | |
| Critical/Blockers Fixed and Verified | Yes | |
| Metrics/Telemetry | ||
| QA Signoff - Nightly Release | Email to be sent | |
| QA Aurora - Full Testing | Yes | |
| QA Signoff - Aurora Release | Yes | Email sent |
| QA Beta - Full Testing | ||
| QA Signoff - Beta Release | Email to be sent |
