Thunderbird/Release Driving/Rapid Release Activities/QA Release Testing

We significantly rely on branch unit tests to ensure build quality. But things can unexpectedly break in automation, in other words all automated test may have passed, and you may have what looks like a "good build", but upon using it turns out to have problems - perhaps even only on one platform. So manual testing (typically smoketests) must be done by QA lead and other testers to ensure the builds correctly install and update correctly in all of our supported environments. QA lead should record testers' results with names of who tested which specific OS versions and locales, so if a problem gets detected at a later date we can go back and say, "but we tested that, what went wrong?"

Special steps

• QA lead download a Windows build and submit it to virustotal.

General Suitability tests

Testing should be done in a "clean" environment, so installs should be done into an empty, new user OS account where Thunderbird has never been installed. (A VM can help.) Also, on Windows the account should NOT have administrator privileges.

1. Do the install into clean environment
2. Mail - add both pop and imap accounts, send an email from pop to imap and send from imap to pop, check you can get+receive new mail and get notified via the OS notification, reply to an email, create a virtual folder based on tags, tag a few emails, see that they appear in the virtual folder
3. RSS - add rss account, add a few feeds (eg http://www.theregister.co.uk/hardware/headlines.atom http://planet.mozilla.org/thunderbird/rss20.xml), check that you can read the feeds and that they get updated
4. news - add a news account at news.mozilla.org and subscribe to mozilla.test (you may use a fake e-mail), post in the newsgroup and reply to a post
5. addons - (optional) test installing and using an addon. If you are familiar with calendar/lighting, please install to see that the proper version installs and is basically usable.

Update testing

During the release process, multiple channels get tested at different steps in the process. (But usability testing above gets done only for one channel.)

Channels

Testing of the update process is performed on test channels. If the buildbot e-mail is "updates available on releasetest", then use the "releasetest" channel. Otherwise, the e-mail should be "updates available on betatest" or "beta-cdntest" in which case use respective channel.

Final releases are not pushed to mirrors until QA signs off, so the sequence is test first on betatest (to sign-off), then releasetest.

Beta and ESR releases automatically get pushed to mirrors. So just be test on releasetest, skipping the betatest channel.

Steps

When update "snippets" are available, test updates on Windows, Linux and MacOS (Linux: for minor betas update tests can be skipped, but should be done betas immediately preceding major releases):

• mix locales (test 2-3 locales _per_ platform)
• mix complete and partial updates (e.g. 12.0 -> 14.0 and 13.0.1 -> 14.0)

For each partial update

1. Download and install the previous version that you're updating from.
2. Edit <apppath>/defaults/pref/channel-prefs.js or channel.js and change "release" to the channel to be tested.
3. Start Thunderbird, and update
   • Help -> About.
   • Verify that the channel is correct in "You are one the XXXXX update channel."
   • Restart when prompted.
4. Check update results
   • the new version starts up
   • a what's new tab is displayed if needed for that release (it may be a 404 page if the page isn't live yet)
   • check version in Help->About
   • check start page (alt+home) wording and all links (release notes, etc)

Testers

After the QA lead has quickly done suitability tests above, the chosen test group should be notified with detailed instructions, in such a combination of people that all three OS and multiple locales are tested. (Also, it is important to have more testers than your minimum test results require - you should recruit twice as many potential testers as needed.) A specific update channel should also tested (except for linux). Give testers an expected time frame for completion and response. You should ask testers to reply inform you if they know they cannot finish testing in the targeted time frame, so that the release process is not delayed and QA can look for an additional tester

Signoff

When responses are received from testers for all 3 OS and sufficient locales, then QA lead emails thunderbird-drivers with signoff confirmation. This should be done as a new message, not as a reply to buildbot or other email, so the signoff confirmation does not get missed by releng and others. Title and contents example "QA signoff Thunderbird Beta 38.0b6 Build2 builds and updates." This will ensure that the signoff email has every needed detail - channel, release#, build# and what was tested - for releng to be confident to move to its next step.