QA/Platform/DOM/Metrics
This page documents the various metrics QA uses to assess the quality of Firefox/Gecko's DOM implementation from version to version. The purpose of these metrics is to help us make informed decisions about testing strategy. These metrics serve several needs, including but not limited to:
- identifying areas of risk for testing before we release a new version of Firefox
- informing root cause analysis discussions when something inevitably slips through the cracks
- measuring the trends in product quality over the long-term
- measuring competitiveness and spec compliance
Please contact Anthony Hughes if you have any questions or feedback regarding this initiative. He can usually be found in #qa on irc.mozilla.org between 09:00 and 17:00 (UTC-9/-10) Monday to Friday by mentioning the nickname ashughes.
Contents
Phase I: Establishing a Baseline
The goal of the initial phase is to establish a baseline that all future quality can be measured against.
In implementation we will experiment with a lot of different metrics and data sources in an attempt to see which metrics, either singularly or in conjunction with other data, enable us to effectively make informed decisions.
Drafted
The following metrics were developed using data from Kyle Lahnakoski's elastic search Bugzilla database. Pending peer review, the next step will be to evaluate better charting and eventually a more automated solution.
- Unresolved crash bugs by component - possibly correlate to KairRo's reports:
- https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.release.35.0.components.html
- https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.beta.36.0.components.html
- https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.aurora.components.html
- https://crash-analysis.mozilla.com/rkaiser/2015-02-05/2015-02-05.firefox.nightly.components.html
-
Unresolved intermittent failure bugs by componentDeprioritized for further investigation following discussion with Andrew Overholt - Unresolved regression bugs by component
- Unconfirmed bugs by component
- Backouts by component
- Regressions resolved prior to Beta (updated as of Firefox 37)
Pending Investigation
Draft proof of concepts can be found in my Google Drive
- https://github.com/AutomatedTester/futurama-data
- backout rate
- http://brasstacks.mozilla.com/testreports/weekly
- tests enabled week over week
- tests disabled week over week
- tests skipped week over week
- https://bugzilla.mozilla.org
- crash bugs over time (draft)
- intermittent failure bugs by age (draft)
- intermittent failure bugs by number of comments (draft)
- intermittent failure bugs by sub-component (draft)
- intermittent failure bugs in/out (draft)
- regression bugs by age (draft)
- regression bugs by number of comments (draft)
- regression bugs in/out (draft)
- regression bugs by sub-component (draft)
- bugs with in-testsuite coverage (draft)
- tracked bugs by version (draft)
- unconfirmed bugs by sub-component (draft)
- bugs by platform (draft)
- mean time to find regression window [ask klahnakoski about elastic search db]
- mean time to resolve a bug with a regression window vs a regression without a window [ask klahnakoski about elastic search db]
- to be determined
- spec test coverage
- http://www.tjhsst.edu/~jcranmer/m-ccov/
- https://treeherder.mozilla.org/#/jobs?repo=mozilla-central
- https://scitools.com/ for code complexity analysis, possibly other metrics
- http://metricsgraphicsjs.org/ for JS library to create dashboards
- Telemetry
- FHR
- Socorro
- Release Management
- OrangeFactor