Release:Release Automation on Mercurial:Staging Specific Notes
Release automation in staging is mostly the same as in production, but does have a few differences you should know about:
- Use the config files with the prefix "staging_". These have many values already set correctly for staging.
- All uploading is done to upload.ffxbld.productdelivery.prod.mozaws.net and the files are visible at https://ftp.stage.mozaws.net/pub/firefox/candidates/. Thunderbird uses upload.tbirdbld.productdelivery.prod.mozaws.net for uploads, same downloads from the same host but different candidates directory.
- It can take a few tries to get the repos created properly. This is because hgweb sometimes returns a 500 (internal server error) when we query about a locale. Just retry the failing repos.
- Staging doesn't have any default set of slaves, you'll need to make loan requests. You can prevent your master from running dep builds using limit_branches in master_config.json.
Contents
How to start a staging release
Setup staging repos
You can use the stage-ffxbld repos or your own hg user account for the repos. If you use stage-ffxbld you most likely need to run this script on a build slave with staging ssh keys available. For your own user repos just make sure you have your own key in ssh-agent (or equivalent).
This is what you'd do for testing a X.0b1 release ahead of time:
#!/bin/bash
for repo in compare-locales tools buildbotcustom buildbot buildbot-configs; do
echo "deleting $repo"
ssh hg.mozilla.org edit $repo delete YES
echo "cloning $repo"
ssh hg.mozilla.org clone $repo build/$repo
done
# substitute mozilla-aurora --> mozilla-beta if you want to do a beta from the current cycle, or release before merging from beta
for l in `wget -q -O- http://hg.mozilla.org/releases/mozilla-aurora/raw-file/tip/browser/locales/shipped-locales |grep -v en-US | awk '{print $1}'`; do
echo "deleting locale $l"
ssh hg.mozilla.org edit $l delete YES
echo "cloning locale $l"
ssh hg.mozilla.org clone $l releases/l10n/mozilla-aurora/$l
done
# these may change if you're testing mozilla-release
ssh hg.mozilla.org edit mozilla-beta delete YES
ssh hg.mozilla.org clone mozilla-beta releases/mozilla-beta
Running the merge scripts
You'll need this if you're doing a b1 before we actually merge. This is based on the main docs for migration.
# checkout mozharness from the gecko tree using archiver wget http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py python archiver_client.py mozharness --repo mozilla-central --rev default --debug
Add a patch like this to suppress tagging of the production aurora repo
diff -ru mozharness.base/configs/merge_day/staging_beta_migration.py mozharness/configs/merge_day/staging_beta_migration.py
--- mozharness.base/configs/merge_day/staging_beta_migration.py 2016-02-25 14:32:51.000000000 -0800
+++ mozharness/configs/merge_day/staging_beta_migration.py 2016-02-25 16:44:44.000000000 -0800
@@ -8,8 +8,8 @@
"tools_repo_url": "https://hg.mozilla.org/build/tools",
"tools_repo_revision": "default",
- "from_repo_url": "ssh://hg.mozilla.org/releases/mozilla-aurora",
- "to_repo_url": "ssh://hg.mozilla.org/users/stage-ffxbld/mozilla-beta",
+ "from_repo_url": "https://hg.mozilla.org/releases/mozilla-aurora",
+ "to_repo_url": "ssh://hg.mozilla.org/users/nthomas_mozilla.com/mozilla-beta",
"base_tag": "FIREFOX_BETA_%(major_version)s_BASE",
"end_tag": "FIREFOX_BETA_%(major_version)s_END",
diff -ru mozharness.base/scripts/merge_day/gecko_migration.py mozharness/scripts/merge_day/gecko_migration.py
--- mozharness.base/scripts/merge_day/gecko_migration.py 2016-02-25 14:32:51.000000000 -0800
+++ mozharness/scripts/merge_day/gecko_migration.py 2016-02-25 18:05:34.000000000 -0800
@@ -174,7 +174,7 @@
def query_push_dirs(self):
dirs = self.query_abs_dirs()
- return dirs.get('abs_from_dir'), dirs.get('abs_to_dir')
+ return dirs.get('abs_to_dir')
def query_push_args(self, cwd):
if cwd == self.query_abs_dirs()['abs_to_dir'] and \
# run the script
python2.7 mozharness/scripts/merge_day/gecko_migration.py \
-c mozharness/configs/merge_day/aurora_to_beta.py \
-c mozharness/configs/merge_day/staging_beta_migration.py
Verify changes are as expected with
hg -R build/mozilla-beta diff | less
Then commit and push
python2.7 mozharness/scripts/merge_day/gecko_migration.py \ -c mozharness/configs/merge_day/aurora_to_beta.py \ -c mozharness/configs/merge_day/staging_beta_migration.py \ --create-virtualenv --commit-changes --push
Buildbot configs
When working in staging you'll need to swap out the buildbot-configs, build/tools, buildbotcustom, and compare-locales repos for either stage-ffxbld ones, or your own. If you're using your own, you also need to change some "root" variables that calculated repos are based on. Here's an example of the changes you'll need:
diff --git a/mozilla/config.py b/mozilla/config.py
--- a/mozilla/config.py
+++ b/mozilla/config.py
@@ -34,3 +34,3 @@ GLOBAL_VARS = {
'graph_selector': '/server/collect.cgi',
- 'compare_locales_repo_path': 'build/compare-locales',
+ 'compare_locales_repo_path': 'users/nthomas_mozilla.com/compare-locales',
'compare_locales_tag': 'RELEASE_AUTOMATION',
@@ -2569,3 +2569,3 @@ BRANCHES['mozilla-release']['platforms']
######## mozilla-beta
-BRANCHES['mozilla-beta']['repo_path'] = 'releases/mozilla-beta'
+BRANCHES['mozilla-beta']['repo_path'] = 'users/nthomas_mozilla.com/mozilla-beta'
BRANCHES['mozilla-beta']['l10n_repo_path'] = 'releases/l10n/mozilla-beta'
diff --git a/mozilla/staging_config.py b/mozilla/staging_config.py
--- a/mozilla/staging_config.py
+++ b/mozilla/staging_config.py
@@ -31,4 +31,4 @@ GLOBAL_VARS = {
'staging': True,
- 'config_repo_path': 'users/stage-ffxbld/buildbot-configs',
- 'buildbotcustom_repo_path': 'users/stage-ffxbld/buildbotcustom',
+ 'config_repo_path': 'users/nthomas_mozilla.com/buildbot-configs',
+ 'buildbotcustom_repo_path': 'users/nthomas_mozilla.com/buildbotcustom',
'stage_server': 'upload.ffxbld.productdelivery.stage.mozaws.net',
diff --git a/mozilla/staging_release-fennec-mozilla-beta.py.template b/mozilla/staging_release-fennec-mozilla-beta.py.template
--- a/mozilla/staging_release-fennec-mozilla-beta.py.template
+++ b/mozilla/staging_release-fennec-mozilla-beta.py.template
@@ -6,3 +6,3 @@
# you're starting a release without Release Kickoff. You have been warned.
-EMAIL_RECIPIENTS = []
+EMAIL_RECIPIENTS = ['<nthomas@mozilla.com>', ]
@@ -58,3 +58,3 @@ releaseConfig['sourceRepositories'] = {
releaseConfig['l10nRelbranch'] = None
-releaseConfig['l10nRepoPath'] = 'users/stage-ffxbld'
+releaseConfig['l10nRepoPath'] = 'users/nthomas_mozilla.com'
releaseConfig['l10nRevisionFile'] = 'l10n-changesets_mobile-beta.json'
@@ -63,4 +63,4 @@ releaseConfig['l10nJsonFile'] = r
releaseConfig['otherReposToTag'] = {
- 'users/stage-ffxbld/compare-locales': 'RELEASE_AUTOMATION',
- 'users/stage-ffxbld/buildbot': 'production-0.8',
+ 'users/nthomas_mozilla.com/compare-locales': 'RELEASE_AUTOMATION',
+ 'users/nthomas_mozilla.com/buildbot': 'production-0.8',
}
@@ -94,3 +94,3 @@ releaseConfig['ausServerUrl'] = '
releaseConfig['doPartnerRepacks'] = False
-releaseConfig['partnersRepoPath'] = 'users/stage-ffxbld/partner-repacks'
+releaseConfig['partnersRepoPath'] = 'users/nthomas_mozilla.com/partner-repacks'
releaseConfig['partnerRepackPlatforms'] = ()
@@ -133,3 +133,3 @@ releaseConfig['single_locale_options'] =
'--tag-override', '%s_RELEASE' % releaseConfig['baseTag'],
- '--user-repo-override', 'users/stage-ffxbld',
+ '--user-repo-override', 'users/nthomas_mozilla.com',
'--cfg', 'single_locale/staging.py',
@@ -141,3 +141,3 @@ releaseConfig['single_locale_options'] =
'--tag-override', '%s_RELEASE' % releaseConfig['baseTag'],
- '--user-repo-override', 'users/stage-ffxbld',
+ '--user-repo-override', 'users/nthomas_mozilla.com',
'--cfg', 'single_locale/staging.py',
@@ -158,3 +158,3 @@ releaseConfig['multilocale_config'] = {
'--tag-override=%s_RELEASE' % releaseConfig['baseTag'],
- '--user-repo-override=users/stage-ffxbld',
+ '--user-repo-override=users/nthomas_mozilla.com',
'--pull-locale-source',
@@ -167,3 +167,3 @@ releaseConfig['multilocale_config'] = {
# Staging config
-releaseConfig['build_tools_repo_path'] = "users/stage-ffxbld/tools"
+releaseConfig['build_tools_repo_path'] = "users/nthomas_mozilla.com/tools"
releaseConfig['enableSigningAtBuildTime'] = True
diff --git a/mozilla/staging_release-firefox-mozilla-beta.py.template b/mozilla/staging_release-firefox-mozilla-beta.py.template
--- a/mozilla/staging_release-firefox-mozilla-beta.py.template
+++ b/mozilla/staging_release-firefox-mozilla-beta.py.template
@@ -6,3 +6,3 @@
# you're starting a release without Release Kickoff. You have been warned.
-EMAIL_RECIPIENTS = []
+EMAIL_RECIPIENTS = ['<nthomas@mozilla.com>', ]
@@ -72,3 +72,3 @@ releaseConfig['sourceRepositories'] = {
releaseConfig['l10nRelbranch'] = None
-releaseConfig['l10nRepoPath'] = 'users/stage-ffxbld'
+releaseConfig['l10nRepoPath'] = 'users/nthomas_mozilla.com'
releaseConfig['l10nRevisionFile'] = 'l10n-changesets_mozilla-beta'
@@ -76,4 +76,4 @@ releaseConfig['l10nRevisionFile'] = '
releaseConfig['otherReposToTag'] = {
- 'users/stage-ffxbld/compare-locales': 'RELEASE_AUTOMATION',
- 'users/stage-ffxbld/buildbot': 'production-0.8',
+ 'users/nthomas_mozilla.com/compare-locales': 'RELEASE_AUTOMATION',
+ 'users/nthomas_mozilla.com/buildbot': 'production-0.8',
}
@@ -164,3 +164,3 @@ releaseConfig['bouncer_submitter_config'
# Misc configuration
-releaseConfig['build_tools_repo_path'] = "users/stage-ffxbld/tools"
+releaseConfig['build_tools_repo_path'] = "users/nthomas_mozilla.com/tools"
releaseConfig['enableAutomaticPushToMirrors'] = True
Tools
Linux update verify will not obey certain settings from release.py. To work around this, make a change like the following to scripts/release/updates/chunked-verify.sh:
diff --git a/scripts/release/updates/chunked-verify.sh b/scripts/release/updates/chunked-verify.sh --- a/scripts/release/updates/chunked-verify.sh +++ b/scripts/release/updates/chunked-verify.sh @@ -24,5 +24,5 @@ if $JSONTOOL -k properties.NO_BBCONFIG $ fi; if [ -z "$NO_BBCONFIG" -a -z "$BUILDBOT_CONFIGS" ]; then - export BUILDBOT_CONFIGS="https://hg.mozilla.org/build/buildbot-configs" + export BUILDBOT_CONFIGS="https://hg.mozilla.org/users/nthomas_mozilla.com/buildbot-configs" fi if [ -n "$PROPERTIES_FILE" -a -f "$PROPERTIES_FILE" ]; then
Setup Ship It
A proper staging release should be done through a staging instance of Ship It regardless of whether or not you're making changes to it. There's two parts to doing this, both of which can be run on dev-master01 and are described below.
The web app
First off, you'll need an instance of the ship it web application to submit your release to. It's recommended to run this on dev-master2. To get it running, start a screen or tmux session and run the following:
git clone https://git.mozilla.org/build/release-kickoff.git cd release-kickoff virtualenv venv venv/bin/pip install -r requirements/dev.txt -r requirements/compiled.txt venv/bin/python kickoff-web.py -d sqlite:///kickoff.db -u shipit -p shipitgood --host=0.0.0.0 &> log &
If you get any errors about ports being in use you can use the "--port" option to choose a different one.
Once that's running, test it out by browsing to http://dev-master2.bb.releng.use1.mozilla.com:5000/submit_release.html (or the port you specified) to make sure it's working. The default username/password are shipit:shipitgood (from the kickoff-web.py cmdline above)
Release runner
You'll also need release runner, which will process releases submitted to your ship it instance and kick them off on your master.
You should also make sure you have mq and rebase enabled in your hgrc, in case release runner needs either of them. Eg:
[extensions] mq= rebase=
Your dev master also needs to be set-up correctly. Make sure the following is true:
- <your_master>/Makefile should be a copy of buildbot-configs/Makefile.master, not a symlink. You need to comment out the call to update-master-json.py in the update target (this will fail because your dev master won't appear in the production master list).
- <your_master>/{buildbot-configs,buildbotcustom,tools} must be Mercurial repositories (NOT Git). Release runner will be pushing and pulling from them.
Release runner needs a few configuration files to function correctly, below are samples from bhearsum. Note that the json file is the same content as the one created by the buildbot master Makefile, but encased in "[]". Also note that the repo paths in release-runner.ini are _local_ paths, not http:// - this is important.
- release-runner.ini:
[api]
api_root: http://dev-master2.bb.releng.use1.mozilla.com:6710
username: shipit
password: shipitgood
[release-runner]
notify_from: Release Eng <DONOTreleaseME@mozilla.com>
notify_to: nthomas@mozilla.com
smtp_server: localhost
hg_host: hg.mozilla.org
hg_username: nthomas@mozilla.com
hg_ssh_key: dummy
buildbot_configs: https://hg.mozilla.org/users/nthomas_mozilla.com/buildbot-configs
buildbot_configs_branch: production
buildbotcustom: https://hg.mozilla.org/users/nthomas_mozilla.com/buildbotcustom
buildbotcustom_branch: production-0.8
tools: https://hg.mozilla.org/users/nthomas_mozilla.com/tools
tools_branch: default
masters_json: nthomas-master.json
sendchange_master: dev-master2.bb.releng.use1.mozilla.com:9710
staging: true
verbose: true
sleeptime: 30
ssh_username: nthomas
[symlinks]
production_config.py: {buildbot_configs}/mozilla/localconfig.py
thunderbird_production_config.py: {buildbot_configs}/mozilla/thunderbird_localconfig.py
- nthomas-master.json:
[
{
"basedir": "/builds/buildbot/nthomas/build1",
"bbconfigs_branch": "production",
"bbconfigs_dir": "/builds/buildbot/nthomas/build1/buildbot-configs",
"bbcustom_branch": "production-0.8",
"bbcustom_dir": "/builds/buildbot/nthomas/build1/buildbotcustom",
"buildbot_bin": "/builds/buildbot/nthomas/build1/bin/buildbot",
"buildbot_branch": "production-0.8",
"buildbot_python": "/builds/buildbot/nthomas/build1/bin/python",
"buildbot_setup": "/builds/buildbot/nthomas/build1/buildbot/master/setup.py",
"buildbot_version": "0.8.2",
"datacentre": "scl1",
"db_name": "dev-master2.bb.releng.use1.mozilla.com:/builds/buildbot/nthomas/build1/master",
"enabled": true,
"environment": "staging",
"hostname": "dev-master2.bb.releng.use1.mozilla.com",
"http_port": 8710,
"master_dir": "/builds/buildbot/nthomas/build1/master",
"limit_branches": [],
"limit_tb_branches": [],
"limit_b2g_branches": [],
"limit_projects": [],
"limit_branch_projects": [],
"mobile_release_branches": ["mozilla-beta"],
"name": "nthomas",
"pb_port": 9710,
"release_branches": [
"mozilla-beta",
"mozilla-release"
],
"role": "build",
"ssh_port": 7710,
"thunderbird_release_branches": ["comm-esr38"],
"tools_branch": "default",
"tools_dir": "/builds/buildbot/nthomas/build1/tools",
"queuedir": "/builds/buildbot/nthomas/common-queue"
}
]
You'll need to run release runner from somewhere with your SSH key in your keychain. Eg, run it from your laptop or forward the the key to dev-master2 If you're forwarding the key, you should connect to dev-master1 as follows:
ssh-add ~/.ssh/id_rsa ssh -A dev-master2.bb.releng.use1.mozilla.com
When you're ready, you can run release runner with the following commands:
hg clone http://hg.mozilla.org/users/yourldap_mozilla.com/tools cd tools/buildfarm/release # default branch is release runner for promotion, use the old branch hg up old-release-runner virtualenv venv venv/bin/pip install buildbot==0.8.2 fabric simplejson PATH=venv/bin:$PATH venv/bin/python release-runner.py -c /path/to/your/release-runner.ini &> log &
If everything is working correctly you should output similar to the following:
2013-03-04 05:54:24,047 - DEBUG - Fetching release requests
2013-03-04 05:54:24,168 - INFO - No new releases: {'releases': []}
2013-03-04 05:54:24,168 - DEBUG - Sleeping for 30 seconds before polling again
Start the release
Using your staging Ship It instance, submit a release! Make sure you use your own repository for "branch" instead of the production one.
Known Issues
Update verify fails
When mar signature verification is enabled on each platform update verify starts to break. This is because we use the dep cert in the staging release, but try to apply updates using the updater which expects the release cert. bug 997732 covers this.
Windows was broken from Firefox 30.0 (if not earlier), Mac from 40.0, Linux from 42.0.
To work around, manually trigger the "ready for release" builder after all update verify builders have completed.
