Thunderbird:Testing/Automated Update Testing

The automated update testing suite is based on MozMill and will automatically test updates for multiple revisions on specified update channels.

Setting up

(this will need to be done for each platform).

Install MozMill as per this page
- You may also need to easy_install Mercurial (standard8 has problems with this on windows).
Create an empty directory, e.g. update-tests

mkdir update-tests
cd update-tests

hg clone http://hg.mozilla.org/qa/mozmill-automation/

Get the script to download the builds from bug 580237, and create the download directory for the builds

wget -O download.py --no-check-certificate \
  https://bugzilla.mozilla.org/attachment.cgi?id=459821
chmod 755 download.py
mkdir builds

You are now ready to run the tests.

Make sure you are in the update-tests directory
Download one or more builds to test update from, these do not have to be the same version (e.g. 3.0.x and 3.1.x builds to test together would be fine)
- Firefox typically test en-US plus two to four other locales for each point release.

./download.py -v 3.1 -p mac -l en-US

Parameters: -v <version> -l <locale> -p <platform> Note: Platform is one off win, mac, linux

./mozmill-automation/testrun_update.py --channel=release \
  --application=thunderbird builds

Parameters: --channel is the main one to change here.

Once the update tests are complete, you'll get a pass or fail summary for each test, including:
- What was updated to/from
- Locale
- Build IDs

The builds downloaded into builds/ are left unchanged, so you can test against them multiple times, e.g. for different update channels (releasetest, release etc).
The tests use their own profile
The automation code downloads a second repository that contains the actual javascript files for running the mozmill tests, that code can be found here.