« QA/Team

This is an overview of what we do from a July 2007 presentation.

QA Team structure and members

• Tools dev - robcee, alice, coop
• Automated test development - Martijn (mw22), bc, ctalbert
• Test Execution - marcia, tracy, juanb, tomcat, tchung, abillings, stephend
• Other: Community-jay, Analysis-ss, Accessibility-tkeenen

Concepts

• Developers handle unit tests
• Create automated functional/integration/system tests where possible
• Use human eyes and ears where necessary
• Right tool for the job - depends on stage of the project and nature of the thing being testing

Test Execution

• Products covered: FFx, TBird, customer extensions or branding (kodak, ebay, world cup)
• Releases covered: Major releases, security releases, partner builds, AMO/WebDev,
• Activities:
  • bug verification
  • en-US smoke tests, BFT, FFT
  • L10n
  • focused testing/exploratory testing
  • Updates
  • Top sites
  • Extension testing

Analysis

• bug triage, troubleshooting, investigation
• Hendrix
• Reporter
• crashers - Talkback/Breakpad
• UNCO - unconfirmed bugs triage- graph

Test Development

• Manual tests in litmus
• Automated tests
  • from bugs
  • for new features
  • eggplant tests
  • Test Tools (see below)

Test Tools

• Tools we have developed for QA use or general use.
• Litmus
• RefTest
• Mochitest
• XPCShell
• Buildbot execution Infrastructure
• Talos - performance framework
• Minotaur - preferences checking tool for L10n and Partner builds
• L10n Metadiff
• Update Checker
• Spider

Community

• QMO
• Test days
• Bug Days
• Mailing lists
• Work with Universities - ex: Seneca College and Club Moz