QA/Firefox3/Strategy

From MozillaWiki
Jump to: navigation, search

« QA/Firefox3

QA Test Strategy – Firefox

Overview

This is a tracking document that outlines the test strategy that Mozilla QA will follow regarding a thorough test pass of Firefox 3. It will follow the general plan on steps and process that QA will take to accomplish a "sign off" to the release strategy. Mozilla QA strives to uphold quality software, and a successful pass corresponds to no blocking issues, 'Green'-colored test pass on test cases in both execution and automation, and a thorough bug verification window of all Blocking bugs and Critical bugs.

NOTE: Updates to this document continue to change as Mozilla continues to refine development process.

Areas Covered

Areas not Covered

  • Feature Unit Tests
    • These are normally covered by feature developers, not QA. However, we do have a Javascript test suite that will collectively find bugs with in-testsuite javascript test cases that Bob Clary will run on occasion. Need more definition here on what QA/Dev should own.
  • String Localization Tests
    • Many of these tests are owned by localizers and SmartWare, our third party testing company. See more in the L10n Test Deliverables section.
  • In depth functionality of Plugins, Extensions, and Themes
    • Many extensions are hosted on AMO, but are not developed and tested by mozilla. We rely on our volunteers to manage these third party extensions. Mozilla QA will focus on high level Addon management, error handling, and a few top extensions supported. But extension functionality will not be covered by Mozilla QA.

Schedule

The release schedule is broken up by milestones, determined by the Firefox team. Each milestone will contain feature enhancements, bug fixes, and updates. Most testing of focused areas and regression areas will come in the latter half of the schedule, during the latter parts of alpha up until the final release candidate. The following test schedule depicts how QA will be breaking down the test cycles.

Questions: Are there plans to branch? If so, what would the schedule look like?

Test Tools

This is ongoing set of Test Tools used by the QA team for automation and execution of the Firefox Product. Some of these tools are developed in-house, and are only accessible within the internal Mozilla network. The following list of tools shown here contain a short description of the tool, as well as further explanation on how the relevant tools are used for specific Test Areas below.

  • Litmus Test Case Manger -
    • Manages a suite of test cases used for smoketests, BFTs, FFTs, regression, and functional areas. Public usage.
  • JS Test Suite -
    • JavaScript project engine. Code name: Spidermonkey. Owned by Bob Clary and written in JavaScript, used to test different javascript API's and functions.
  • Security Test library -
    • Bob Clary finds ways to test security issues: automates ways to throw noise at "fuzz testers"; triggers randomness to hit edge cases. The way we use it, we target security issues. (eg. take HTML structure of webpage, look at tags in HTML, and find cases to execute within the tags.) Ways to exploit memory, leaks, execution. Tries to automate crashing tests by generating HTML and js code to crash the browser.
  • Download Checker -
    • Bob Clary again. Takes a huge list of locales and OS installs. highly error prone. Script will go to webpage, download each build, installs, runs a test, and closes it. fully automated. This is executed only after we go live. (all.html - "other system and languages")
  • L10n search verifier - This has been replaced by Minotaur tool. (see below)
    • Don't want to break localized products that may have mistakes. Search engines, bookmarks, RSS feeds that are pre-configured. Idea is to catch changes before it's checked in. Diffs the region.properties settings for each locale against rules defined in the l10n requirements. (http://wiki.mozilla.org/Firefox2/L10n_Requirements)
  • Talos - Performance Testing Project
    • Talos is a performance testing project. With a framework written in Python it runs Ts (startup test) and Tp (page load test) while monitoring memory and cpu usage. Ts is a simple Javascript web page that times the loading and then exits. Tp consists of a Javascript web page that cycles through a set of locally served web pages. The web page set is a collection of the top 500 web pages as listed by Alexa; they have been 'cleaned' so that all of their content is locally served.
  • Eggplant -
    • Graphical replay; Tracy owns it. Litmus smoke tests on it. Identify a spot, string a list of actions, and play back.
  • MochiTests -
    • Built on the Mochikit platform. Main developer - robert sayer. unit test framework, easy way to run the browser. Scripting actions to quickly create test cases and tie to executables using JavaScript. Also supports accessibility layers.
  • BuildBot -
    • test system for building. owned by Robcee. client / server interaction. Client requests execution activities. Server sets tests up, run results. (eg. parallel make system for builds. able to build pieces independently and can piece it together later) Like a next-gen tinderbox.
  • Reftests -
    • Unit test framework, easy to run in the browser, easy to write. The testcases don't rely on external files. Mainly useful for layout tests, only.
  • Minotaur -
    • Minotaur is a testing framework that can automatically check the preferences, search settings, bookmarks, extensions, and update channels for a Firefox build. These settings are vetted against a previous gold standard of settings and/or verification files. This is used to simplify and reduce the manual testing currently required for Localization and Partner builds of Firefox.

Categories

Feature Tests

Summary

Our feature testing approach is to match major features to test engineers and have the test engineering work with the development engineers during development and as the feature lands to ensure the features are well tested.

Planning & Scheduling

Each milestone marker in the Firefox scheduling, will contain a list of features that are noted and checked into the tree for the release. The QA Execution team will be dividing up the Firefox 3 features and developing a test plan for each module. Test plan will be under initially reviewed by peers, and posted to newsgroups for further review. Completion of testing the feature is targeting the final Beta in the cycle.

Developers focus on unit tests. QA will focus on areas where the unit test burden is very onerous and focus on user scenarios, API testing, integration testing exploratory testing, and system level testing. QA will utilize this Test Case Template as a formatting tool to create test cases on these features. Both automation and manual test cases will be created and stored within litmus, bugzilla, and CVS.

We will track the design, landing, test plan development, test case development, and running of the test cases for the initial feature testing via this tracking spreadsheet.

Current drafts of the Firefox 3 Feature Test Plans are listed here under "Test Plan Location" column.

The expectation is that the major features are well tested during the Alpha phase or maybe B1 at the latest. Then testing during later betas is focused on verifying bug fixes, testing additional edge cases that are identified (polish testing), and regression testing using smoke tests, BFTs, FFTs and automated runs.

Test Framework

Automation:

  • Test cases within Bugzilla Bugs - some testcases are Unit tests, API-level, and can only be executed within a development environment.
  • Mochikit
  • Reftests
  • Browser Chrome
  • XPCShell
  • JProf

Execution:

  • Litmus (BFTs, FFTs, smoketests, Focused Tests)
  • Test cases within Bugzilla Bugs - some testcases are written and executed within feature bugs that are done manually.
  • Eggplant - a set of functional tests ran and results reported to litmus
Results

Test Results for executed test cases will be tracked within the litmus tests. For automation results, unit tests that are created in mochikit will be ran on tinderboxen. Other test results

Web Compatibility Tests

Summary

The Web Compatibility Tests focuses on regression and compatibility with popular websites and web applications.

Planning & Scheduling

We will be gathering the most relevant and widely used web applications today. Web compatibility tests will be broken down by new feature tests for Firefox 3 specifically, as well as regression across the top extensions, plugins, and themes posted on AMO. Tests will be expected to run during the later stages of alpha all the way up to RC2.

Test Framework
  • Top Sites - coverage of a few popular sites that drives most user traffic
  • Top extensions - coverage of Top firefox extensions used by the public
  • Top themes - coverage of Top firefox themes used by the public
  • Top plugins - coverage of Top web plugins used by the public
  • Protocol handling - regression of a popular list of protocols supported on the web. (eg. mailto://, https://, ftp://) Ties into litmus content handling test cases.
Results

Test results are tracked in execution of FFTs, BFTs, and smoketests in Litmus.

User Performance Tests

Summary

The User Performance Tests are not to be confused with Browser performance tests that are run with automation. Instead, we will concentrate on day to day usage of Firefox and any performance issues that we see. These would includes areas like downloading, idling, and other Firefox daily usage cases. It will also incorporate some failover scenarios that would add a few stress cases and error handling cases.

Planning & Scheduling

Different scenarios will be conducted and posted on litmus or a public wiki. Since these frameworks involves a different mix of computing environments, we will ask for community assistance to help. The timing can exist around the Beta schedule, for about 1 or 2 cycles.

Test Framework
  • Performance against other browsers? Do we have benchmark comparisons?
  • Hard drives, throttling CPU’s
  • Internet connection (wireless, dial up, LAN)
  • Tab switching test
  • Fail-over error-handling testing scenarios in Litmus
Results

Test cases will exist in litmus and the public wiki. Any issues found will be tracked in Bugzilla.

Configurations Tests

Summary

The necessity to simulate multiple OS environments and profiles is needed to capture different scenarios and issues that may occur on one environment but not the other. This test suite will focus on running common Litmus smoketests and BFTs against a combination of environments in a lab setting and/or a virtual machine.

Planning & Scheduling

QA is developing virtualization test suites that contains images of various profiles across XP, vista, and Linux virtual machines. This will help to assure a clean environment for testing, as well as cross functional platform operation. The Environments TestPlan will be divided up among the team and run during the Beta stages of the overall schedule.

Test Framework
  • Virtual Machines environment
  • Mixed Profile settings (see marcia’s link to all the different profile combos)
  • Platforms specific (mac, vista, linux, xp)
Results

Issues found will be tracked in Bugzilla, and test results will be recorded in Litmus. Virtual Machines are tracked on the file server and available for download with VMWare licensing.

Security Tests

Summary

Security tests are critical for catching external attacks through Firefox to the user. Mozilla is constantly finding and issuing fixes to malware, leaks, and identity attacks to assure the user of the safeguard of Firefox. The goal of these tests are to attempt to catch as many new attacks as possible, as well as regress past security issues that have been fixed but broken again.

Planning & Scheduling

QA will be creating new tests for new security features added to Firefox 3. Tests will be run in the beta to RC stage, and a test plan will identify the strategy. There are also security cases automated within Eggplant that cover simple security checks like cookie manager, website passwords, etc.. Development has a series of unit cases and fuzzer tools that they will run on individual environments.

Test Framework
  • Browser Security tests at Scanit
  • Bob Clary's security tests - Javascript cases created and automated on Eggplant.
  • Platforms with anti virus software, firewalls, global passwords, vista parental controls
  • Jesse’s Fuzzer Tools - jsparsefuzz.js
  • Security bugs in Bugzilla
Results

Test results will be tracked via automation through eggplant and jesse's fuzzer tools. There are a few security tests also tracked in litmus to be ran manually. Also a handful of security bugs in Bugzilla have test cases to be run independently. (Example: Bug 389106)

Accessibility Tests

Summary

Accessibility Tests will be developed and ran against the new features coming within Firefox 3. These cases will ensure we are following proper A11y guidelines.

Planning & Scheduling

Testing of Accessibility features will be within the Alpha to Beta stage of the schedule. TimK will be heading up the feature testing (current Draft), but also rely heavily on community participation. There are also plans to put in more automation around a11y software to run through test scenarios.

Test Framework
  • Accessibility areas against new features
  • Regression of areas, manual/ automation
Results

Test cases will exist in Litmus and the public wiki. Any issues found will be tracked in Bugzilla.

Bug Verifications

Summary

New code is landing every night into the tree, and current functionality is constantly being update and overridden. Mozilla uses Bugzilla to track all incoming and outgoing bugs, from code fixes to design mockups.

Planning & Scheduling

To prioritize and narrow the list of bugs, QA will focus primarily on Firefox bugs that represent highest severity, and carry the "blocking" flag. This is more so a "goal" to verify every blocking and priority 1 bug, and will be the benchmark to "ship" or "stop" the release.

Test Framework
  • Process to watch bugs on nightly channel, landing per feature
  • Cross watching other team member’s bugs
  • Dedicated time frame to work only on bug verifications during a milestone
  • Updating Litmus and reftests based on bugs
Results

Bugs will be marked verified or reactivated for regression. All results tracked in Bugzilla.

Distribution Tests

Summary

Distribution tests concentrate on the various configurations and locales of Firefox that are customized to specific partner builds. Many of the changes will be configuration based, but each partner will have a special branding and set of tests to be ran in conjunction of a release. This is to preserve the validation and regression of the product.

Planning & Scheduling

New test cases will be developed for new distribution requirements. Current regression tests documented in the Partner Test Plan, will be ran during the later beta stages of the release.

Test Framework
  • DEX Requirements (new configurations)
  • Partner Major/Minor Updates
Results

Results are tracked on the intranet wiki page and reported to the support group for Firefox partners.

L10N Tests

Summary

The L10N Tests will prepare a set of standard Firefox testing requirements. Its purpose is to ensure that third party localizers are properly checking the validity of the build before checking in to the tree. It also will provide a quick automated check of configurations and values, and report results.

Planning & Scheduling

String freeze for Firefox 3 will be targeting a time frame between B1 and B2. Test Execution will be developing a few test cases that they determine would be important to be checked during localization testing in their main Firefox 3 features. A customized l10n test suite will be created for the localizer to run on private builds prior to checkin.

The prioritized list of locales for Firefox 3 is listed here.

Test Automation will be running through a tool that will effectly validate customizations and configurations.

A third party company, SmartWare, is also contracted to verify strings and spotchecks on selected locales. The localizer is also responsible for unit tests prior to checkin. Click to see the existing L10N Test Plan.

Test Framework

Automation

  • Minotaur

Execution

  • Standard litmus test suite for localizer to run
  • Work with 3rd party Smartware to identify test cases
  • Spot checks on Tier 1 locales
Results

Test results are tracked in different forms. The localizer will track their unit cases and litmus for the custom l10n test suite. Test execution team will report results in Litmus. Test automation will track results at <insert location here>. Smartware will also track results at <insert location here>.

Updates Tests

Summary

Update Testing is the process of verifying minor security patches and major version releases from a prior version. The purpose of this Test Suite is to validate existing behavior, as well as spot check assortment of locales, strings, AUS2 interaction, and download links.

Planning & Scheduling

Update testing comes after a minor or major build has landed and ready for verifications. It does not include nightly and hourly updates on the trunk. Responsibilities are divided up by locales and OS's, and ran through a thorough list of configuration checks as well as smoketests and spot checks. It is also checked against the beta server (staging) and production (bouncer). Update testing occurs no earlier than beta 2 stage and is tested on ever release thereafter.

Test Framework
  • Minor updates (partial, failed)
  • Major updates (complete)
  • Spot Checks, Configurations
Results

Results are reported via wiki and email. There are also live updates tests against the production server after it has been released.

Collecting Feedback

There are multiple ways we collect comments and feedback for Firefox releases. The goal of feedback helps QA track issues found in development, regression, and post-release analysis.

Reporting tools

Regression Tests

Regression Tests are constantly executed over the course of a release. The purposes of these tests are to plant cross-platform overlapping tests, to ensure stability and reliability of the builds post nightly checkins. The QA Tiger Team was formed for this reason, to ensure nothing is easily missed. See the detailed description and schedule Here.

Manual Tests

Manual Regression tests are ran weekly and at every milestone marker. The majority of the test cases are recorded in litmus and available for anyone to execute. The QA Tiger team has been executing weekly test runs on Smoketests, BFTs, and focused areas. Any of these tests can be publicly executed and reported against Litmus.

Manual framework
  • Smoketests - A subset of litmus tests ran to ensure the quality of a nightly build. Executed daily.
  • Basic Functional Tests - A more comprehensive subset of litmus tests that cover more regression than smoketests, but not a full suite. Executed once a week.
  • Full Functional Tests - A thorough set of litmus tests that cover each and every test case that is created to cover full functional regression and feature functionality. Executed at least once per release.
  • Eggplant Cases - An automated independent tool that takes the smoketests across all platforms and executes them on demand.
  • Focused feature areas across supported Firefox platforms
  • Feature review of nightly checkins found on tinderboxen. see Bonsai Query.

Automation Tests

Both Daily and triggered automation cases exist in different frameworks that run a suite of cases like security, performance, and API level test cases.

Automation Framework
  • Buildbot
  • Mochitests
  • Reftests
  • Browser Chrome
  • XPCShell
  • Performance tests
  • Automated Memory Leaks (rob sayer's unit cases)

Results are recorded on tinderboxen. Many of these tests are run automatically on nightly checkins, and will flash orange if tests have failed.

Community Involvement

The goal is to actively involve more community members in testing Minefield, Gran Paradiso, and ultimately Firefox 3 as it moves into beta and to final release.

There are many active participants in the community that are enthusiastic about Firefox and want to contribute to our quality assurance efforts. However, getting them involved and maximizing the value of their time and effort requires a lot of coordination, planning, and enforcing of the correct processes and procedures. Since such activities create constant challenges for our QA team, we are working on a new strategy to promote more community involvement.

Current avenues of community involvement include:

  • Friday Test Days
    • increased the frequency of these events to allow people to show up each week to try out new features and help us run through test cases. Activities include triaging unconfirmed bugs, running tests in Litmus with trunk builds (and alphas/betas), Q&A/Help sessions. We will be focusing on a new Firefox 3 feature/component every week.
  • Tuesday Bug Days
    • a full day each week dedicated to working with various bug lists in Bugzilla. Activities will rotate between unconfirmed bug triage, bug verifications, test case writing, and reproducing crash bugs.
  • QMO - quality.mozilla.org
    • Using the website to promote news and articles on what the Mozilla QA team is working on
    • Provide events and forums for the community to get involved.
    • Create closer relationships with regular contributors and help them become leaders for our events and projects.
    • Promote the QA Extension to get more people connected to our tools and processes. We will be setting up a "product page" on QMO and start the process for getting it polished enough for AMO.
    • Update/migrate outdated QA pages on mozilla.org and showcase them on QMO
  • Campus Evangelism - Mozilla University
    • Leverage the Campus Reps program to reach out to students and professors at schools.
    • Reach out to folks interested in open source and help them discover Mozilla project and get involved in Firefox development and testing.
  • Better "Beta" product pages and programs
    • Update Minefield and Gran Paradiso pages to direct folks to QMO and other useful channels for getting involved and providing feedback.
    • Coordinate community testing with marketing/pr plans for Firefox 3 Beta release
Retrieved from "https://wiki.mozilla.org/index.php?title=QA/Firefox3/Strategy&oldid=79576"