Overview

Excerpt

Mozmill tests allow us to have a much broader coverage of manual tests as we ever could have when running tests by hand. The speed-up is incredible and allows us to run a dozen or even more test-runs in the same time as you would be able to do by hand. Therefore we are building up a complex scenario of tests from all possible areas, which will be run against all available builds of Firefox, on each platform, and in any locale.

This project covers the work which is needed to have such a complex test-suite in-place.

Project Details

In general there are about 700 manual tests on Litmus which could be automated. Those tests are separated into three different groups: Smoketests, BFT (Basic Functional Tests), and FFT (Full Functional Tests). While seeing that amount of tests are focus is currently on Smoketests and BFT tests. The reason therefore is that those tests have to be run for every security release of Firefox and need attention from QA. Having those automated would give us the chance to set the focus of manual testing to areas where check-ins happened for the new release. All other tests could be run automatically.

Once we have nearly finished the BFT testgroup, we will have to check if automating all FFT's is useful or not.

Finding a Test to Write

1) Learn how to run Mozmill Tests

• If you haven't done so already, you should learn how to run Mozmill tests on your own computer. This is a crucial first step in understanding what a Mozmill test is, how it is structured, and how it works. It's also a great point of reference when writing your own tests.

2) Check the Milestone

• Look above at the current milestone. This will help you identify where we need the most help. Ideally we would like you to help with our current milestone.

3) Check the Spreadsheet

• Look at our Google spreadsheet of test automation. Scroll down to find the area matching our current milestone. All Mozmill tests are correlated to a Litmus manual test case. In general, starting from the simplest test is a good idea.

4) Still a bit Lost?

• If you need help finding a test to work on, feel free to get in contact with the project lead, Anthony, either by email or on irc.mozilla.org in the #mozmill channel (nick: ashughes).
• Anthony will help you identify a test, guide you with getting started in test development, and push forward any issues (like Shared Module development) which may prevent you from writing your test; or if he can't he'll find someone who can.

Preparing to Write

Now that you've found a test, it's time to start writing your test. Before you begin, there are a couple things you should do to prepare.

1) Read the Documentation

• Before you begin writing your test you should familiarize yourself with the Mozmill Tests Style Guildelines. Following these guidelines will streamline the review process of your patches.

2) File a Bug

• The next step is to get a bug report on file to track development of the test. Simply follow the template and some simple rules in filing your bug:
  • Refer to the name of the Litmus test in the bug summary; something like "Mozmill test for pinning an App Tab"
  • Give the bug a description about the Litmus test; something like "Tracking bug for developing a Mozmill test for pinning an App Tab"
  • Provide a link to the most recent Litmus test in the URL field
  • Mark the bug as ASSIGNED and put your email address in the ASSIGNEE field
  • Update the bug number field in the spreadsheet, mark the status as "assigned", and add your name.

3) Ready, Set, Go

• Now you are ready to begin writing your first test

Writing Your First Test

1) Know the Guidelines

• In case you missed it, or need a refresher, be sure you are familiar with the specifics of our Mozmill test writing guidelines.

2) Templates

• We've created a test skeleton template which should help you with getting started. Simply copy and paste the content of the template into a new file; this will give you the raw structure needed for a Mozmill tests. Make sure you fill in the license plate with your name and email address.

3) The Litmus Test

• When writing your Mozmill tests, you should follow the steps and expected results of the Litmus test as closely as possible. In some cases variation and deviation from the Litmus test may be desired or even warranted.

4) Shared Modules

• An integral part of our Mozmill tests API is the Shared Modules. These make it easier to implement your test by doing a lot of the getter, setter, and helper legwork required for many tests. Using the shared modules also makes it easier for others to understand what your code does. We strongly encourage familiarization with the shared modules.

Landing Your First Test

1) Review

• Before we can land your test into the mozmill-test repository you will have to create a patch and go through our review process. Reviewing patches is an iterative process of submitting a patch and asking experienced developers to check the code for style, completeness, functionality, and stability. Every patch which lands in a Mozilla repository undergoes a review.

2) Super Review

• Once your patch passes review, known as r+, it will be handed off to one of the project leads. The purpose of the super-review is to catch any last minute flaws which may have been missed during the initial review phase. It's a quick stop along the way to ensure the quality and stability of the test repository. Every patch which lands in the Mozmill Tests repository undergoes a super-review.

3) Check-in

• Once your patch has been given r+ by both the reviewer and super-reviewer, it will be checked in across various branches of the Mozmill-tests repository. In general, there is nothing for you to do at this stage but to wait for someone on the team to check it in for you. You can take satisfaction in a job well done and the appreciation of Mozmill and the community.

Be aware that any test you contribute will be your responsibility to fix should it begin to generate errors during our daily testruns

You are now ready to write your next test -- simply repeat the process from the beginning (it get's easier with each new test)