QA/Automation/Projects/Mozmill Automation
Overview
Mozilla QA is using the existing Mozmill tests to run automated tests against any kind of Firefox build. While this is reducing the time we have to spent on manual testing, we can also enhance our testing to more than only Firefox BFT tests. We have automated test-runs available for software update tests, l10n tests, add-on tests, and others. Those are getting run on any platform, and also for localized builds of Firefox.
Name: | Mozmill Automation |
Leads: | Henrik Skupin |
Contributors: | n/a |
Etherpad: | http://etherpad.mozilla.com:9000/qa-mozmill-automation |
Sub Projects
Sub Project | Status | Description |
Mozmill CI | Active | Perform fully automated Mozmill testruns via Jenkins CI |
PuppetAgain | Active | Puppetizing Mozmill CI machines |
Mozmill Environment | Active | Supply a pre-configured environment for Mozmill tests execution |
Mozmill CI Configuration Generator | Active | Application to generate valid configuration files for Mozmill CI on-demand jobs |
|
Finished | Update automation scripts for Mozmill to support Mozmill 2.0 |
|
Finished | |
|
Finished | |
|
Finished | |
Details
The way to reach the state when everything is automated is long can can be divided into the following sub-projects:
- Setting up a machine which can be used for automated tests
- Creation of scripts which cover all automation aspects
- Running the automation for release builds
Machine for Automated Testing
A machine in the QA lab is needed at least for the first time. Once we have a stable process we can consider to integrate the automation into the Releng process.
This sub-project has been finished. All requirements can be read on bug 537840
Automation Execution
Right now Mozilla QA is using the machine in the QA lab to run release tests. All tests have to be triggered manually. We consider to use Hudson in the future to control the complete process.
Automation Scripts
The automation scripts should allow us to have a complete automated process in-place. That means that only a trigger would be necessary to start those release tests. The following areas have to be covered:
- Script for Downloading builds from the FTP Server (bug 528064)
- Script for Smoketests, BFT, and FFT Test-runs (bug 563523)
- Script for Software Update Test-runs (bug 564539)
- Script for Add-ons Test-run (bug 562445)
- Script for L10n Test-run (bug 565196)
- Script for Accessibility Test-run (bug 565201)
Downloading builds from the FTP Server
For a complete automation of our Mozmill tests for release testing, a download script is necessary which will fetch all the builds to test from the FTP server and store those locally. The work is being tracked on bug 528064.
Smoketests, BFT, and FFT Test-run
To execute all existing Mozmill tests which have been written for the Smoketest, BFT, and FFT testgroup of Litmus this script is useful. Once it has been called it should automatically execute the steps below:
- Install a build (if requested)
- Run the tests from all those three testgroups
- Uninstall the build (if requested)
The work is being tracked on bug 563523.
Software Update Test-run
To run the software update tests automatically a test-run script is required which will take care of any aspect of the tests to be run. Once it has been called it should automatically execute the steps below:
- Install a build (if requested)
- Create a backup of the build
- Run the partial or complete software update tests
- Restore the backup
- Run the fallback software update tests
- Restore the backup
- Run background software update tests
- Uninstall the build (if requested)
A script with this feature set has already been created on bug 504653 and is able to execute all the steps except step 6. The reason is that we do not have any automated tests for background software updates yet.
Add-on Test-run
The script for the add-ons test-run simplifies the work which would have to be done to run Mozmill tests against available add-ons. Once it has been called it should automatically execute the steps below:
- Install a build (if requested)
- Perform Mozmill tests against any available add-on
- Download the specified version of the add-on
- Start Firefox with the extension installed
- Run all available tests
- Clean-up the system
- Uninstall the build (if requested)
The work is being tracked on bug 562445.
L10n Test-run
With the creation of l10n tests we will be able to perform specific tests for localizers. Once it has been called it should automatically execute the steps below:
- Install a build (if requested)
- Run all the l10n tests
- Uninstall the build (if requested)
The work is being tracked on bug 565196.
Accessibility Test-run
With the creation of accessibility tests we will be able to perform specific tests for keyboard navigation. Once it has been called it should automatically execute the steps below:
- Install a build (if requested)
- Run all the accessibility tests
- Uninstall the build (if requested)
The work is being tracked on bug 565201.