QA/IntersectionObserver
Overview
Purpose
This wiki details the testing that will be performed by QA for the Intersection Observer API. It defines the overall testing requirements and provides an integrated view of the project test activities. Its purpose is to document:
- Areas of risk
- What will be tested
- How testing will be performed
- criteria
- ownership
- schedule
To help ensure the best possible release and long term maintenance of this feature.
Risk Analysis
Risk area | Requirement | Status |
---|---|---|
Ad intensive web pages | No regression in interaction performance | PASS on Nightly 54 |
Security | Fuzz testing finds no security holes | PASS on Nightly 54 |
Stability | Crash rate does not increase | TBD |
Testing Strategy
Scope
Test Areas | Covered |
---|---|
Private Window | Yes |
Multi-Process Enabled | Yes |
Multi-process Disabled | Yes |
Stability | Yes |
UI | |
Display (HiDPI) | Yes |
Interraction (scroll, zoom) | Yes |
Add-ons | |
ad-blocking Add-ons must be disabled/removed to test | Yes |
Security | |
Fuzzing | Yes |
Web Compatibility | |
Testing against target sites | Yes |
If it's not listed above, it is currently out of scope.
Objectives
Criteria Description | Metric | w/o IO API | w/ IO API | Criteria Met? | QA Owner |
---|---|---|---|---|---|
Manual testing | Interaction performance | Pass | Pass | Yes on Nightly 54 | Tracy |
Unit testing | Automated tests | Pass | Pass | Yes on Nightly builds of 54 | Tobias |
Fuzz testing | Automated tests | Pass | Pass [1] | Yes on Nightly 54 | Fuzzing team (Raymond) |
Stability | Telemetry/Crash reports | tbd - telemetry experiment in bug 1341259 | tbd - telemetry experiment in bug 1341259 | soon to begin with 10% enabled | Tobias |
[1] tracking bug has no bugs hanging off of it
Testcases
Ad-hoc testing of Ad-intensive websites. In general news sites tent to be rich of Ads. Therefore the top news sites on Alexa are a great source for suitable websites.
[1] The top 500 news sites on the web
List of known sites using the Intersection Observer API:
- https://soundcloud.com/charts/top
- http://news.yahoo.co.jp
- http://www.iza.ne.jp/kiji/entertainments/newslist/entertainments-n1.html
- http://www.bbc.com/sport/all-sports
- http://money.cnn.com/video/media/2016/10/13/rogue-one-trailer-good-morning-america.cnnmoney?iid=EL
- http://money.cnn.com/2016/11/25/media/rogue-one-tickets/
- http://abc7news.com/pets/hillsborough-mountain-lion-goes-viral/1658809/
Possible test scenarios:
- Open a single news site. Wait till it is fully loaded to make sure all Ads are properly initialized. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
- Open a multiple news sites in different tabs. Wait till every page is fully loaded to make sure all Ads are properly initialized. Switch between tabs from time too time. Close and open new tabs randomly. If there is content to scroll, scroll up and down a couple of times. Click on some articles to trigger a refresh.
- Repeat scenarios above but let the browser untouched for a longer period of time. Then proceed.
Environments
Full Testing will be performed using Nightly for Desktop builds on:
- Windows
- Mac OS X
- Linux
Full Testing will be performed using Nightly for Mobile builds on:
- Android
Channel dependent settings (configs) and environment setups
- In Nightly (and DevEd when available)- dom.IntersectionObserver.enabled needs to be set to "true" in about:config
- If any kind of Ad-blocker is installed/in use it needs to be uninstalled/disabled
Manual Testing Results
Ad-hoc testing around testcases and scenarios above in regular and private windows, in e10s and non-e10s windows and on normal and HiDPi displays.
Nightly
Platform | IO disabled | IO enabled | QA'd by : build ID |
---|---|---|---|
Windows | Pass | Pass | Tracy : 20170228030203 |
Mac | Pass | Pass | Tracy : 20170228030203 |
Linux | Pass | Pass | Tracy : 20170301110155 |
Android | Pass | Pass | Tracy : 20170301110150 |
Aurora
Platform | IO disabled | IO enabled | QA'd by : build ID |
---|---|---|---|
Windows | (results) | (results) | TBD |
Mac | (results) | (results) | TBD |
Linux | (results) | (results) | TBD |
Android | (results) | (results) | TBD |
Beta
Platform | IO disabled | IO enabled | QA'd by : build ID |
---|---|---|---|
Windows | (results) | (results) | TBD |
Mac | (results) | (results) | TBD |
Linux | (results) | (results) | TBD |
Android | (results) | (results) | TBD |
Project Information
Ownership
Developer contacts: Tobias Schneider
QA: Tracy Walker
QA Peer: TBD
Builds
This section should contain links for builds with the feature
Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 20160128 | tbd |
Study documentation/specs received from developers | 20170201 | 20170210 |
QA - Test plan creation | 20170210 | 20170215 |
QA - Nightly Testing | 20170227 | 20170301 |
Telemetry - Stability Experiment | 20170304?? | tbd |
QA - Aurora Testing | tbd | |
QA - Beta Testing | tbd | |
Release Date | tbd |
References
- Meta bug
- Bug 1321865 - Enable IntersectionObserver