Beta Schedule

Calendar

Week 1

• Monday: Merge Day, Beta 1 is built
• Thursday: Beta 1 is shipped after QA signs off the builds

Week 2

• Tuesday: Beta 2 is built
• Wednesday: Beta 2 is shipped after QA signs off the builds
• Thursday: Beta 3 is built
• Friday: Beta 3 is shipped after QA signs off the builds

Week 3

• Tuesday: Beta 4 is built
• Wednesday: Beta 4 is shipped after QA signs off the builds
• Thursday: Beta 5 is built
• Friday: Beta 5 is shipped after QA signs off the builds

Week 4

• Tuesday: Beta 6 is built
• Wednesday: Beta 6 is shipped after QA signs off the builds
• Thursday: Beta 7 is built
• Friday: Beta 7 is shipped after QA signs off the builds

Week 5

• Tuesday: Beta 8 is built
• Wednesday: Beta 8 is shipped after QA signs off the builds
• Thursday: Beta 9 is built
• Friday: Beta 9 is shipped after QA signs off the builds

Week 6

• Tuesday: Beta is merged to Release, RC is built
• Wednesday: RC is pushed to the Beta channel following QA sign-off

Week 7

• Monday: RC is pushed to the Releasetest channel following QA sign-off
• Tuesday: RC is pushed to the Release channel following QA sign-off

Beta Walkthrough

Merge Day

The following tasks should be completed on or before merge day:

• Pre-Beta Feature Sign-offs:
  • feature owners will send an email detailing their "pre-beta sign-off status"
  • if you don't see this email by merge day, send an email to each feature owner to ensure they're ready
  • contact release drivers if any features are not ready for beta and need to be disabled
• Moztrap Regression Testrun:
  • Create a moztrap regression testrun to be run throughout the Beta cycle
  • Include tests based on past regressions
  • Include tests based on new features
  • Include tests based on bugzilla components with a high-volume of high-risk changes
• Mozmill Tests Migration
  • the automation team will merge the mozmill-tests branch to mozilla-beta
  • this needs to be done before Beta builds are generated, otherwise we may get a lot of false failures in Mozmill Tests

Beta Builds

Each Beta builds needs to have the following completed:

• Pulse-triggered Mozmill Tests
• On-demand Mozmill Tests
• Betatest Channel Update Tests
• Releasetest Channel Update Tests
• Stability spotcheck
• Support spotcheck
• Sign-off
• Beta Channel Update tests

Mozmill Functional Tests

There are two types of Mozmill functional tests: pulse triggered and on-demand.

• Be sure you are subscribed to mozmill-ci@mozilla.org so you can see failure emails

Pulse Triggered Automation

These tests are triggered automatically as soon as builds become available. These test basic functionality in the builds from our top-10 locales. For the purposes of Beta build sign-off you only need to check the dashboard to make sure there wasn't a high volume of failures and that any failures that did occur are reported in Bugzilla.

On-demand Automation

These tests are the same as the pulse triggered tests except that they need to be triggered by hand. The on-demand tests give pair-wise coverage to the locales which were not tested via pulse. In other words, we ensure that all locales have been tested at least once before signing off.

1. Create a plain-text ondemand_functional_firefoxN.cfg config file (example)
2. Connect to the VPN
3. Load the trigger interface
4. Upload your config file and click Build
5. Click the Back to Dashboard link, you should start to notice your tests running on the nodes on the left.
6. Review the dashboard for any failures (note: it will take a couple of hours before final results are posted)

On-demand Update Automation

Note: Firefox 28 and earlier update to 29.0b8 -- do not include anything earlier than 29.0b1 in your config

Beta -> Beta

• Wait for an email from [release] indicating "updates are ready"
• Create a config file by copying a previous config file and updating the version numbers (we usually test a few previous versions)
  • be sure the config file indicates the update channel you want to test
  • be sure the config file indicates the correct target build ID
• Connect to the VPN
• Load the trigger interface
• Click "Browse" and locate/select your config file
• Click "Build"
• Click the Back to Dashboard link, you should start to notice your tests running on the nodes on the left. It will take a couple of hours before final results are posted.
• Load the results dashboard and filter on the version you tested
• Additionally filter on a TO date of the day you ran the tests and a FROM date of the day before
• Additionally filter on the update channel you tested
• Review the results for any failures and search Bugzilla for related bugs if failures are reported
• This query can be helpful, listing open mozmill-test failures
• If bugs exist add a comment including links to failure reports -- file a new bug if one does not exist

RC -> Beta (example config)

• We also need to test updates from the previous RC to the current Beta
• Repeat the same steps as above with a slightly different config file
  • add override-update-channel=firefox-mozilla-beta to the [testrun] section
  • use the current Release as your source version
• after this finishes, do it again but with the channel=releasetest and name the file accordingly

Beta Cycle

The following needs to be completed throughout the Beta cycle, either as part of or in parallel to a build sign-off.

• Regression testing
• Bug fix verifications
• Incoming bug triage
• QAWANTED bug triage
• Feature testing