User:Wsm/MilestoneTesting
Thunderbird Release Candidate Milestone Testing
Overview
Release Candidate builds for Alpha, Beta and ... provided by builders must be tested by volunteers and the QA team according to a rigorous process in order to certify that a release candidate build is worthy for public release.
We have just a few days to complete testing, so we need all available hands. Thank you for your interest, and if circumstances allow, your enthusiastic participation. As always, help is available on IRC and in the newsgroups for both new and experience testers.
We have several types of tests (most of which utilize litmus), and you can choose which to run based on your available time. BFT testing and l10n, and Mac platform tests are our greatest need.
- BFT (Basic Functionality Tests) - touches most functional areas. If you do not have time to do BFT tests we hope that you will ...
- Smoketest - touches a handful of functional areas (a much reduced BFT)
- Spot check - a visual "look see"
- l10n (localization) - A set of litmus tests just for localized builds. If you can test a localized version of Thunderbird, we would appreciate you doing so.
If you can't help before the scheduled testing period ends, please stop back and run the tests as soon as you are able.
You are not required to run a full test set to completion. So, if you don't have much time or think you might run out of time that's OK. In fact, we'd rather have you do a short amount of good testing than none at all. Tests done using litmus get recorded as you go through the list, so stop whenever you need ... and return later when (and if) you are able.
Details
Important
- Use only the builds specified in this message. Do not use a nightly build. Do not use a previous alpha or beta.
- Please file a bug whenever you encounter a problem.
- If a litmus testcase:
- Is not run by you (you skipped it) - please do not mark it in any way with a comment, passing or failed.
- Fails - please comment in the test, file a bug, and include URL of testcase in bug
- Is unclear - please comment with steps and wording that you think would make it more clear
- QA/Thunderbird3/TestResults/Alpha3 documents the release details. You want to check sections 1.1 to 1.6, especially the release notes. Contact methods are also listed.
Builds
Builds are listed in the release details. Use a localized version if possible. Please do not use old alphas, betas or a nightly.
BFT
Basic Functionality Tests is organized into subgroups listed below. It is a large list, but to shorten your testing time (running the full set takes 4-6 hours) you can focus by subgroups as follows:
- * indicates priority items - if you have the resource required please do * groups before others
- All indicates items we hope everyone will test
- AL and MZ - indicates please run that group if your last name/surname is in the range A-to-L or M-to-Z
If you start running out of time, or you get done with the assigned subgroups, please do more subgroups, especially those that show the least amount of testing coverage as shown on the "Select Subgroup" list.
Getting started:
- You've already read the release notes, of course.
- login to litmus.mozilla.org (create an account if you don't have one)
- visit https://litmus.mozilla.org/run_tests.cgi?test_run_id=34
- complete all the information about your setup - paste buildid, set platform, OS, and locale (note: there is only one linux OS selection)
- click "submit configuration"
- You're off and running the tests! Please try * items first. Figure 2.5-3.5 hours to do half the list (as assigned)
Subgroup assignments for BFTs All-Install, shutdown, uninstall MZ-Migration (from other mail systems) MZ-Import (from other mail systems) MZ-Window configuration All-Toolbars and menus All-Account settings & Preferences (Options) * IMAP accounts * POP accounts (exclude Global Inbox) * Global inbox All-Mail composition MZ-Spell checker MZ-RSS account & subscriptions MZ-Newsgroups All-Navigating and displaying messages AL-Downloading and saving AL-Image blocking AL-Return receipts * Proxies MZ-Offline, disk space All-Moving, copying, deleting messages All-Views and labeling messages MZ-Message filters MZ-Message search AL-Address search AL-Saved Search folders AL-Message Grouping All-Message Aging All-Quicksearch MZ-Address books MZ-Junk mail MZ-Extensions AL-Theme management AL-Help AL-Printing AL-Master Passwords & password management * Phishing, spoof detection * Secure connections * Digital signing, encrypting messages AL-Software Update
smoketest
Access litmus using the procedure above for BFT, but use the smoketest URL: https://litmus.mozilla.org/run_tests.cgi?test_run_id=33. Should take about 1.5 hours. (Ignore "Subgroup assignments" listed above - it is not relevant)
Spot check
Spot check takes the least time because is just a casual visual inspection and does not require the use of litmus. Install, poke around, inspect, test and try what you can, then visit http://wiki.mozilla.org/QA/Thunderbird3/TestResults/Alpha3#Thunderbird_Spot_Check and update the wiki.
locale
You can do any of the tests as described above in your native language build. Then please update the wiki at Thunderbird3/TestResults/Alpha3/L10N. This will help us keep track of what locales have been tested.
In addition to (or instead of) the above, you can run the l10n trunk baseline tests that focus on checking that localized language strings are correct. The results are recorded in litmus.