Regression Testing

Article to collect here all information about regression testing of Mozilla's products.

Related Mozilla's Products

• Thunderbird

Regression

When you notice that suddenly something breaks with a new build of Thunderbird, and you go back to an older Thunderbird build, and it works again, then you found a "regression". These are bugs that are introduced by a very recent change and what developers are generally very interested in fixing (or at least they should be).

However, to do anything, we need a "regression range", and we need your help for that. The idea is to find a 1-day range of changes that broke things for you, which limits the amount of changes to look at. For that, you need to look at the nightlies and test a number of them.

How to find a regression range using nightly builds

1. Find the last nightly that works and first nightly that breaks. Do a so-called "binary search", e.g. try March 1 (works) and March 30 (fails), then the middle, i.e. March 15 (works), then again the middle, i.e. March 22 (fails), March 18 (works), March 20 (fails), March 19 (works). Now you know that it broke between March 19 and March 20.
2. Look in the FTP download directories of these 2 nightly builds, at the TXT file with the build date and hg commits, and post the contents of the 2 TXT files in the bug.
3. (optional) Look at the hg commits between the to builds, by going to e.g. <http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=121b3a2363c0&tochange=e94edfdb1f5b>, and post that URL in the bug. Do the same with comm-central.
4. (optional) Skim over any commits that might be relevant to the bug. Be sure to "Expand" the merge commits, esp. from inbound. If you find anything, post it in the bug (with disclaimers, if you're not a developer), and assign the bug to the person that made the change.
5. In the bug's keyword field, add "regression"