QA/Desktop Firefox/Walkthroughs/Release Coordination
The following document describes the work flows and processes that guide testing of releases of Firefox for PC platforms, aligned to Release Management's release coordination process.
Nightly
Firefox starts on mozilla-central, this is where the initial code is landed to deliver new features to users. Every 6-8 weeks (consult the Rapid Release Calendar for more information) the current Nightly version is uplifted to the Aurora branch then Nightly becomes the next Firefox version. This "uplift" always happens on a Monday, except when people are unavailable due to statutory holidays.
Timeline
The following describes the general timeline for the entire six week life-cycle of a typical Nightly.
- Day 1:
- Release Management updates the mozilla-central branch to the new version
- QA Lead creates initial test plan to track important bugs and new features
- QA Lead signs off initial Nightly builds
- Interim:
- Developers land changes, fixes, and new features
- QA tests these changes, fixes, and features as they land
- QA Lead ensures important features and bugs are escalated and assigned
- Day 42:
- QA signs off features and builds as ready for Aurora by prior to migration
- Release Management closes the tree and merges mozilla-central to mozilla-aurora
Priorities
- Ensuring topcrash bugs are being investigated
- Ensuring iteration developed bugs are owned and signed off
- Ensuring new features are owned and on track for sign-off
- Ensuring fixed bugs are being verified
- Ensuring reported bugs are being triaged
- More information: walkthroughs, example test plan.
Aurora
The Developer Edition, DevEd (formerly known as Aurora, and still named aurora in many contexts).
Timeline
- Day 0 (Monday)
- Release Management will migrate mozilla-central to mozilla-aurora [example changeset] and sends an email to release-drivers to throttle updates on the aurora channel. Release engineering files a bug (example: bug 1062352
- QA Driver changes Mozmill-CI Automation to use the auroratest channel (mozmill-ci)
- Automation Development migrates the mozmill-tests default branch to mozilla-aurora
- Release Engineering will throttle updates on the aurora channel and sends an email to release-drivers
- QA confirms Aurora updates throttled on the aurora channel and sends an email to release-drivers
- Day 4 (Friday)
- QA confirms no failures in Mozmill-CI functional testruns [example results] and updates for Aurora on the auroratest channel, email Release Drivers to sign-off unthrottling Aurora updates
- Release Engineering unthrottles Aurora updates and sends an email to release-drivers
- QA confirms Aurora updates unthrottled on the aurora channel and sends an email to release-drivers
- QA Driver changes Mozmill-CI Automation to use the aurora channel (mozmill-ci)
- Day 5..38
- QA continues testing their features and bugs with an eye to Beta quality
- QA conducts daily investigations into the most critical QAWANTED bugs
- Day 39 (Friday)
- QA feature owners sign-off their features for migration to Beta by emailing the QA release driver and any developers involved
- Day 42 (Monday)
- Release Management migrates mozilla-aurora to mozilla-beta
Priorities
Migration Day
On the Monday before every release will be Migration Day. Sometime before the end of the day, Release Management will close the mozilla-aurora tree and merge in the changes from mozilla-central. Once this is complete the tree will be reopened and Release Management will file a bug to disable updates on the aurora channel. Once this has been completed QA needs to complete the following tasks:
- Open mozmill-ci, set CHANNEL to use the auroratest channel and click SAVE (necessary to allow our daily automation to continue seemlessly until updates are re-enabled)
- Manually verify that Aurora builds to not update past Monday's Aurora build on the aurora channel
- Manually verify that Aurora builds update to the latest Aurora on the auroratest channel
- Comment in the bug to confirm updates have been disabled and mark the bug VERIFIED FIXED
Sign-off
Following the migration, QA has four days to test and sign-off Aurora builds before updates are enabled. The following tasks should be performed before the Friday following the release.
- Ensure feature owners have tested and signed off their features in Aurora
- Ensure any blocker/critical bugs targeting this milestone have been verified fixed
- Ensure the daily Mozmill functional automation has not flagged any unexpected failures
- Ensure the daily Mozmill update automation on auroratest has not flagged any unexpected failures
- Do a manual spotcheck of the builds and updates on auroratest for increased confidence, as you see fit
Once you have the confidence of the builds and updates, communicate your sign-off to Release Drivers. This should be completed by Friday morning if at all possible.
Release Day
On the Friday following every release updates to Aurora builds will be re-enabled on the aurora channel. After QA has given their OK to enabled the updates, Release Management will file the bug, Release Engineering will push the changes, and QA needs to verify the changes.
- Manually verify that Aurora builds update to the latest Aurora on the aurora channel
- Open mozmill-ci, set CHANNEL to use the aurora channel and click SAVE
- Comment in the bug to confirm updates have been re-enabled and mark the bug VERIFIED FIXED
Prior to Beta
On the Monday prior to release Aurora will be migrated to Beta. This happens without QA sign-off so make sure the following tasks are completed well in advance:
- Feature owners have tested and signed off their features as being "ready for Beta"
- Bugs and stability issues have been triaged and escalated to Release Management
- Ensure all blockers/critical issues have been verified fixed
- Ensure there are no unexpected failures in the daily Mozmill automation testruns
Once the migration happens, the first Beta build will be generated. Testing of this build can begin on Tuesday after the releases have been shipped and must be completed by Thursday. See the Beta section for more details.
Beta
Timeline
General notes:
- Check mozmill configuration files with http://www.mozqa.com/mciconf/app/ before uploading them.
- After go-to-build, it is about 3.5 hours until linux builds out, 5 hours for all en-US builds, 7 hours for all builds, 7.5 hours for updates on beta-localtest.
- No need to run ondemand functional tests any more as of Oct. 2014. All locales are covered in the automated tests that run overnight.
- Mozmill tests run on builds on several update channels.
- beta-localtest tests updates on our ftp server.
- beta-cdntest hits the CDN. (We only need to run one, preferably cdntest)
- beta also tests the update builds on our CDN.
For the releases (as of Jan. 2015):
- Initial testing of updates will be done on "release-localtest" (used to be "betatest").
- Post-mirror push testing will be done on "release-cdntest" (used to be "releasetest").
- Email 'please push to beta channel' after beta-cdntests and regression tests are done, for betas 2-9. QA can authorize the push for beta 2-9. relman authorizes it for beta 1, RC, and release. *** update the walkthrough below with this for each release) ***
- Day 0 (Monday)
- Release Management will migrate mozilla-aurora to mozilla-beta
- Automation Development migrates the mozmill-tests mozilla-aurora branch to mozilla-beta
- Day 1 (Tuesday)
- Release Management will send the go-to-build email for the first Beta
- Release Engineering generates the builds (takes several hours; overnight)
- QA develops a test plan including manual regression tests and bug fix verifications, then coordinates overnight testing of the Beta candidates via email to Softvision.
- Day 2 (Wednesday)
- functional tests can start as soon as 'Updates available on beta-localtest' email comes in. That indicates all the builds are done. (This is obsolete)
- QA runs Mozmill ondemand update tests on thebeta-cdntest channel.
- QA coordinates further overnight testing of the Beta candidates via email to Softvision
- Day 3 (Thursday)
- QA signs off the Beta candidate for release to beta testers via email to Release Drivers
- Release Management sends email to push the Beta updates live
- Release Engineering pushes the updates live and sends an email to Release Drivers when complete
- QA runs Mozmill-Ondemand update automation on the beta channel and sends a sign-off email to Release Drivers when complete
- Day 4..7
- QA continues feature testing, dogfooding, and QAWANTED bug investigations, providing status updates of top issues in the Tuesday & Thursday Release Coordination meeting
- Day 7 (Monday)
- Release Management will send the go-to-build for the first Beta (late in the day?)
- Release Engineering will generate the builds (takes several hours)
- QA develops a test plan including manual regression tests and bug fix verifications, then coordinates overnight testing of the Beta candidates via email to Softvision
- Day 8 (Tuesday)
- QA runs Mozmill-Ondemand functional automation, then update automation on the beta-cdntest channel. Report back on this to release-drivers.
- Release Management sends email to push the Beta updates live
- QA then runs update tests on the beta channel and reports back to release-drivers.
- Day 9 (Wednesday)
- Continue manual testing, as well as monitoring and reporting on crash fixes.
- Day 10 (Thursday)
- Release coordination sends the go-to-build. This will take a few hours for builds to complete, usually overnight : 3.5 hours until linux builds out, 5 hours for all en-US builds, 7 hours for all builds, 7.5 hours for updates on beta-localtest.
- QA coordinates further overnight testing of the Beta candidates via email to Softvision
- Day 11 (Friday)
- QA runs Mozmill-Ondemand update automation on the beta-localtest and beta-cdntest channels.
- QA signs off the Beta candidate for release to beta testers via email to release-drivers
- Release Management sends email to push the Beta updates live
- Release Engineering pushes the updates live and sends an email to Release Drivers when complete
- QA runs Mozmill-Ondemand update automation on the beta channel and sends a sign-off email to Release Drivers when complete.
- Day 11..14
- QA continues feature testing, dogfooding, and QAWANTED bug investigations, providing status updates of top issues in the Tuesday & Thursday Release Coordination meeting
- Day 15 (Tuesday)
- Release Management will send the go-to-build for the third Beta
- Release Engineering will generate the builds (takes several hours)
- QA develops a test plan including manual regression tests and bug fix verifications, then coordinates overnight testing of the Beta candidates via email to Softvision
- Day 16 (Wednesday)
- QA runs Mozmill-Ondemand update automation on the betatest channel, and update automation on the releasetest channel
- QA coordinates further overnight testing of the Beta candidates via email to Softvision
- Day 17 (Thursday)
- QA signs off the Beta candidate for release to beta testers via email to Release Drivers
- Release Management sends email to push the Beta updates live
- Release Engineering pushes the updates live and sends an email to Release Drivers when complete
- QA runs Mozmill-Ondemand update automation on the beta channel and sends a sign-off email to Release Drivers when complete
- Day 18..21
- QA continues feature testing, dogfooding, and QAWANTED bug investigations, providing status updates of top issues in the Tuesday & Thursday Release Coordination meeting
- Day 22 (Tuesday)
- Release Management will send the go-to-build for the fourth Beta
- Release Engineering will generate the builds (takes several hours)
- QA develops a test plan including manual regression tests and bug fix verifications, then coordinates overnight testing of the Beta candidates via email to Softvision
- Day 23 (Wednesday)
- QA runs Mozmill-Ondemand update automation on the betatest channel, and update automation on the releasetest channel
- QA coordinates further overnight testing of the Beta candidates via email to Softvision
- Day 24 (Thursday)
- QA signs off the Beta candidate for release to beta testers via email to Release Drivers
- Release Management sends email to push the Beta updates live
- Release Engineering pushes the updates live and sends an email to Release Drivers when complete
- QA runs Mozmill-Ondemand update automation on the beta channel and sends a sign-off email to Release Drivers when complete
- Day 25..28
- QA continues feature testing, dogfooding, and QAWANTED bug investigations, providing status updates of top issues in the Tuesday & Thursday Release Coordination meeting
- Day 29 (Tuesday)
- Release Management will send the go-to-build for the fifth Beta
- Release Engineering will generate the builds (takes several hours)
- QA develops a test plan including manual regression tests and bug fix verifications, then coordinates overnight testing of the Beta candidates via email to Softvision
- Day 30 (Wednesday)
- QA runs Mozmill-Ondemand update automation on the betatest channel, and update automation on the releasetest channel
- QA coordinates further overnight testing of the Beta candidates via email to Softvision
- Day 31 (Thursday)
- QA signs off the Beta candidate for release to beta testers via email to Release Drivers
- Release Management sends email to push the Beta updates live
- Release Engineering pushes the updates live and sends an email to Release Drivers when complete
- QA runs Mozmill-Ondemand update automation on the beta channel and sends a sign-off email to Release Drivers when complete
- Day 32..35 (Monday)
- Release Management will send the go-to-build for the final Beta
- Release Engineering will generate the builds (takes several hours)
- QA develops a test plan including manual regression tests and bug fix verifications, then coordinates overnight testing of the Beta candidates via email to Softvision
- Day 36 (Tuesday)
- QA runs Mozmill-Ondemand update automation on the betatest channel, and update automation on the releasetest channel
- QA coordinates further overnight testing of the Beta candidates via email to Softvision
- Release Management migrates mozilla-beta to mozilla-release and sends the go to build the first release candidate
- Day 37 (Wednesday)
- QA signs off the Beta candidate for release to beta testers via email to Release Drivers
- Release Management sends email to push the Beta updates live
- Release Engineering pushes the updates live and sends an email to Release Drivers when complete
- QA runs Mozmill-Ondemand update automation on the beta channel and sends a sign-off email to Release Drivers when complete
- Day 42 (Monday)
- Release Management migrates mozilla-aurora to mozilla-beta
Priorities
The following defines the tasks and work involved in a typical "sign-off". (Another walkthrough for beta here: https://wiki.mozilla.org/QA/Desktop_Firefox/Releases/Walkthrough/Beta)
Developing a Test Plan
All release testplans follow the same basic format:
- Checklist
- Functional Automation
- Manual Testing
- Update Automation
- Bug fix verifications
Once Release Management has sent the go to build you should prepare the test plan. You can see an example of a test plan here. Once you've generated the test plan send an email to Softvision to inform them of the testing required and the deadline to sign-off. Be sure to indicate any high-profile/high-risk areas to focus on.
Here is an example email:
Subject: Firefox 21.0b4 Testing
Message: Release Management has sent the go to build our next Beta.
You should start seeing builds on the FTP server in a few hours.
Once the builds are available please begin testing as indicated
by the test plan[1]. Please pay particular attention to Plugins
as there were several changes to the plugins backend which carry
a moderate level of risk for regression. Our plan is to sign-off
and ship these builds by Thursday morning.
Thank you
1. https://wiki.mozilla.org/Releases/Firefox_21/Test_Plan#Beta_4
Functional Automation
Functional automation can be run as soon as all candidate builds are available (~10 hours from go-to-build). To execute the automation you'll want to upload a manifest to the mozmill-ondemand server. You will want to test en-US plus one random locale per platform.
Here is an example manifest:
[testrun] script=functional report=http://mozauto.iriscouch.com/mozmill-release/ [linux ubuntu 12.04 32bit] platform=linux 21.0#1=en-US pt-BR [linux ubuntu 12.04 64bit] platform=linux64 21.0#1=en-US fi [linux ubuntu 12.10 32bit] platform=linux 21.0#1=zh-TW pt-BR [linux ubuntu 12.10 64bit] platform=linux64 21.0#1=es-MX fi [windows 7 32bit] platform=win32 21.0#1=fi da [windows 7 64bit] platform=win32 21.0#1=da it [windows 8 32bit] platform=win32 21.0#1=it ru [windows 8 64bit] platform=win32 21.0#1=ru es-ES [windows vista 32bit] platform=win32 21.0#1=ro ko [windows xp 32bit] platform=win32 21.0#1=ko ro [mac 10.6 64bit] platform=mac 21.0#1=es-ES de [mac 10.7 64bit] platform=mac 21.0#1=fr es-ES [mac 10.8 64bit] platform=mac 21.0#1=de fr
To execute the testrun you'll want to follow these steps:
* Connect to the MTP VPN * Load http://mm-ci-master.qa.scl3.mozilla.com:8080/ * Once the Jenkins interface loads click the admin tab * Click the green arrow button for trigger-ondemand * Upload the config file you've created (document must be in plain text)
Once uploaded go back to the main dashboard, you should see a bunch of ondemand functional testruns queue up. The testruns will take ~40 minutes to execute. Once they finish, check your mozmill-ci email for any failures.
If something goes wrong, please get in touch with ashughes, davehunt, or whimboo in the #automation channel on IRC.
If everything went okay you can look up the results on the mozmill-release dashboard. Filter by Functional Tests > Reports, Branch matching the version tested, OS matching All, and a date range matching the last 24-hours. You should see a list of results matching the testruns. Check for 0 failures in all rows.
If you have any failures you'll need to dig into those to see if they are known failures or not. You should be able to tell by doing a bugzilla query agains the Mozmill Tests component. Otherwise, ask someone in #automation for assistance.
Update Automation
Very similar to Functional Automation, you'll want to upload a manifest to test updates. The difference here is that you'll want to indicate the update channel, target build ID and source versions you want to test. Wait for the build to be done (the "updates are available" email from buildbot).
Here are the channels you'll need to test:
- Beta: beta-cdntest before sign-off, beta after sign-off
- ESR: esr-cdntest before sign-off, esr after sign-off
- Release:
- beta channel: beta-cdntest before sign-off, beta after sign-off
- release channel: release-localtest and release-cdntest before sign-off, release after sign-off
Test the configuration file with mciconf before you upload it to jenkins and build it.
Look over recent check-ins to see if there are changes in currently shipped locales: http://hg.mozilla.org/releases/mozilla-release/log/default/browser/locales/shipped-locales (And file bugs for mciconf when locales are added or removed)
Here is an example manifest:
[testrun] script=update report=http://mozauto.iriscouch.com/mozmill-release/ target-build-id=20130423212553 channel=betatest [mac 10.6 64bit] platform=mac 19.0b1=en-US de 20.0b1=en-US ja-JP-mac 21.0b1=en-US it 21.0b3=en-US es-ES [mac 10.7 64bit] platform=mac 19.0b2=en-US pt-BR 20.0b2=en-US fr 21.0b2=en-US ru 21.0b3=en-US zh-TW [mac 10.8 64bit] platform=mac 19.0b3=en-US it 20.0b3=en-US ro 21.0b2=en-US de 21.0b3=en-US ko [linux ubuntu 32bit] platform=linux 19.0b4=en-US de 20.0b4=en-US ko 21.0b1=en-US zh-TW 21.0b3=en-US ru [linux ubuntu 64bit] platform=linux64 19.0b5=en-US es-ES 20.0b5=en-US pt-BR 21.0b2=en-US de 21.0b3=en-US it [windows 8 32bit] platform=win32 19.0b6=en-US es-ES 20.0b6=en-US zh-TW 21.0b1=en-US ro 21.0b3=en-US da [windows 8 64bit] platform=win32 19.0b1=en-US fr 20.0b1=en-US it 21.0b2=en-US es-ES 21.0b3=en-US zh-TW [windows 7 32bit] platform=win32 19.0b2=en-US es-ES 20.0b2=en-US zh-TW 21.0b1=en-US fr 21.0b3=en-US da [windows 7 64bit] platform=win32 19.0b3=en-US es-ES 20.0b3=en-US de 21.0b2=en-US ko 21.0b3=en-US ro [windows vista 32bit] platform=win32 19.0b4=en-US de 20.0b4=en-US pt-BR 21.0b1=en-US it 21.0b3=en-US es-ES [windows xp 32bit] platform=win32 19.0b5=en-US fr 20.0b5=en-US ko 21.0b2=en-US de 21.0b3=en-US ru
To execute the testrun you'll want to follow these steps:
- Connect to the MTP VPN
- Load http://mm-ci-master.qa.scl3.mozilla.com:8080/
- Once the Jenkins interface loads click the admin tab
- Click the green arrow button for trigger-ondemand
- Upload the config file you've created
Once uploaded go back to the main dashboard, you should see a bunch of ondemand update testruns queue up. The testruns will take ~40 minutes to execute. Once they finish, check your mozmill-ci email for any failures.
If something goes wrong, please get in touch with ashughes, davehunt, or whimboo in the #automation channel on IRC.
If everything went okay you can look up the results on the mozmill-release dashboard. Filter by Update Tests > Overview, Branch matching the version tested, OS matching All, Channel matching the channel you tested, and a date range matching the last 24-hours. You should see a list of results matching the testruns. Check for 0 failures in all rows.
Add a link to this report and update the results to your test matrix in the test plan.
Update Throttling
Depending on the update strategy upon release it is reasonable to expect we need to test update throttling. If updates are at 100% you should get a background update, 0% you should NOT get a background update; you can use this Moztrap test to check that. If you need to test an intermediary throttling percentage use this Moztrap test instead. Here's a useful throttle rate test script: https://bugzilla.mozilla.org/attachment.cgi?id=8532130
The urls to test throttling should look something like this: https://aus3.mozilla.org/update/1/Firefox/33.1.1/20141113143407/WINNT_x86-msvc/en-US/release/update.xml Where 33.1.1 is an older version of Firefox than the one you want to update to (in this case, we are testing updates to 34.0), and the number after it in the url is the buildID of that older version.
To find an accurate version and its buildID, check the formerly-ftp directory for past release candidates:
https://ftp.mozilla.org/pub/firefox/candidates/.
In this example we are looking at:
https://ftp.mozilla.org/pub/firefox/candidates/33.1.1-candidates/build1/
and then looking for the build ID in one of the info.txt files:
https://ftp.mozilla.org/pub/firefox/candidates/33.1.1-candidates/build1/win32_info.txt
Example of a more recent query for Firefox 46: https://aus5.mozilla.org/update/3/Firefox/46.0/20150320202338/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
Note, occasionally the buildIDs may be different for different platforms, so it is good to make sure you check the correct info.txt file for the platform you're listing in the throttling test url.
Rel eng's throttling test page: https://wiki.mozilla.org/Release:Release_Automation_on_Mercurial:Updates_through_Shipping#Verifying_Throttling
This will only need to be done for updates on release channel and only at Release Management's request.
Extra note on how update pings work. Instances on the release channel will send an update ping about every 24 hours. The time of the last check is saved over multiple sessions. So you won't send an update ping on every restart of Firefox. Only if the last update check was over a day ago. The timing is not exact to the minute.
Manual Testing
Manual testing is typically performed by our Softvision team in Romania. What needs testing in a given build is guided by our Beta Checklist. Additionally there may be requirements to check high profile features or high risk bug fixes. There's no real "how-to" here, just use your best judgement.
Bug Fix Verifications
Bug fix verifications are guided by bug queries and mercurial pushlogs. There's no real "rule" for what should and shouldn't be tested. You'll need to read the bugs and assess whether the change is significant enough to warrant testing. The basic litmus test is if the landed patch carries high risk (usually indicated in the uplift approval request) or if the bug is specifically called out in the release notes.
When searching for bugs by query you'll want to search against Target Milestone, the tracking-firefoxN:+ flag, and the status-firefoxN:fixed flag, where N is the version of Firefox you are testing.
When searching for bugs by pushlog you'll want to query between the changeset ID of the current build and the changeset ID of the previous build. For example, the pushlog for Firefox 21 Beta 4 searches from the Beta 3 changeset (e845a10035f2) to the Beta 4 changeset (04aba2e6927f): http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=e845a10035f2&tochange=04aba2e6927f
Any bugs you think warrant verification before a release is shipped should be given the verifyme keyword. Include a query to these verifyme marked bugs and Softvision will test them as part of their sign-off work.
QAWANTED Bug Investigations
QAWANTED bug investigations are ongoing, high priority bugs which need QA testing. For the most part these are not fixed and need regression windows, steps to reproduce, minimized testcases, or URLs so developers can investigate a fix in a timely manner.
Simply querying bugzilla for bugs with the qawanted keyword and the tracking-firefoxN:+ flag set for the current version you are testing should catch most of these. For the most part Release Management will raise these in the Release Coordination meetings or via email, especially if it's critical.
The majority of your effort should be spent trying to nail down these bugs and get them resolved before release. As the release day approaches a judgement call will need to be made to either ship with the known bug or to block the release. While this decision is ultimately Release Management's decision, your judgement of the impact on "quality" is essential.
It is recommended that you track these bugs in the testplan by query and by providing a list of the bugs with their current status and assignee. This list should be revisited on a daily basis.
UNCONFIRMED Bug Investigations
While not necessarily a required part of signing off a release it is good practice to make sure some of the reported bugs have been triaged. Softvision runs a weekly triage session with the community which will typically go through some but not all of these bugs. It is important to triage these bugs to A) move them into the right component so developers are aware, and B) to become aware of serious regressions before we release thus mitigating a chemspill release.
Basically you'll want to triage for UNCONFIRMED bugs with either the version matching your release or the tracking/status flag set to ?. Providing a query to these bugs in your testplan is a good way to get community involved. Any bugs confirmed should be listed in your testplan and escalated to Release Management either by flagging it tracking?, emailing Release Management, or raising it in the Tuesday/Thursday Release Coordination meetings.
Stability Analysis
Stability analysis is fairly straightforward. Simply ask one of the Stability team to review crash-stats and flag any stability concerns related to whichever version you are trying to sign-off. This should be done last, before sending the sign-off to push updates live, to allow for enough time to pass to collect data from early adopters.
Sending the Sign-off
Once you have confidence in the quality of the product you need to send an email to Release Drivers. In general you will need to send two sign-off emails: one to signal pushing updates to the release channel, another to sign-off those live updates. The exception is that Release builds need a conditional sign-off before pushing updates to mirrors (ie. releasetest).
Here are some example emails to illustrate.
- Conditional sign-off
Only applies for Release Candidate builds -- ESR and Beta releases are pushed to mirrors automatically
Subject: [Desktop] QA signs off Firefox 22 for push to mirrors
Message: QA has finished all functional testing and bug fix verifications.
Having found no serious regressions or blockers for release we feel
it is safe to push the snippets to mirrors. Please push snippets at
Release Management's go.
- Release Sign-off
Applies to Release, Beta, and ESR builds
Subject: [Desktop] QA signs off Firefox 22 for release
Message: QA has finished all functional testing, bug fix verifications,
and update tests on releasetest. Having found no serious regressions
nor blockers we feel it is safe to release these builds. Please push
snippets live at Release Management's go.
- Live Update Sign-off
Applies to Release, Beta, and ESR builds
Subject: [Desktop] QA signs off Firefox 22 updates on release channel
Message: QA has finished testing updates to Firefox 22 on the release
channel and everything is working as expected.
Release
Timeline
- Day 0 (Tuesday)
- Release Management migrates the mozilla-beta branch to mozilla-release
- Automation Development migrates the mozmill-tests mozilla-beta branch to mozilla-release
- Release Management sends the go-to-build for the first Release Candidate
- Release Engineering generates the builds (takes several hours)
- Day 1 (Wednesday)
- QA develops a Release test plan including manual regression tests and bug fix verifications, then coordinates overnight testing of the Release Candidates via email to Softvision
- Day 2 (Thursday)
- QA runs Mozmill-Ondemand functional automation, update automation on the betatest channel
- QA coordinates further overnight testing of the Release Candidates via email to Softvision
- Day 3 (Friday)
- QA coordinates further overnight testing of the Release Candidates via email to Softvision
- Day 6 (Monday)
- QA sends conditional sign-off of the Release Candidates via email to the Release Drivers
- Release Management sends the go to push snippets to mirrors
- Release Engineering pushes Release Candidate snippets to mirrors and sends email to Release Drivers when complete
- QA runs Mozmill-Ondemand update automation on the releasetest channel
- Day 7 (Tuesday)
- QA sends final sign-off for release via email to the Release Drivers
- Release Management sends the go to push snippets live
- Release Engineering pushes Release Candidate snippets live and sends email to the Release Drivers when complete
- QA runs Mozmill-ondemand update automation on the release channel then sends a sign-off email to Release Drivers when complete
- Release Management pushes web content
- WebQA verifies web content is live
- RRRT creates a page to track potential .point release issues
- Day 8+
- QA assists with investigating any potential point release issues and prepares test plans for point releases as required
- Day 41 (Monday)
- Point of no return for any point releases
- Day 42 (Tuesday)
- Release Management migrates mozilla-beta to mozilla-release and sends the go to build the next Release
Priorities
ESR
Timeline
- Day 0..36
- QA focuses on verifying most critical and security fixes using the nightly builds
- Day 37 (Thursday)
- Release Management sends the go-to-build for the ESR Candidates
- Release Engineering generates the ESR builds (takes several hours)
- Release Management informs the Enterprise mailing list to start conducting internal integration testing
- Day 38 (Friday)
- QA runs Mozmill-Ondemand functional automation, update automation on the esrtest channel and update automation on the releasetest channel
- QA coordinates Sunday night testing of the ESR candidates via email to Softvision
- Day 41 (Monday)
- Softvision completes all of their requisite testing
- Day 42 (Tuesday)
- QA sends final sign-off for release via email to the Release Drivers
- Release Management sends the go to push snippets live
- Release Engineering pushes ESR snippets live and sends email to the Release Drivers when complete
- QA runs Mozmill-ondemand update automation on the release channel then sends a sign-off email to Release Drivers when complete
