SeaMonkey:Release Process:2.1b2

« SeaMonkey 2.1b2

Build Harness

SeaMonkey:Release Automation

Bugs

Tracking bug filed as bug 631278

Build Engineer

Justin Wood (Callek)

Signed-off Revisions

Build 1

http://hg.mozilla.org/comm-central/rev/8b9494bc2d2d
http://hg.mozilla.org/mozilla-central/rev/f9d66f4d17bf (GECKO20b11_2011020209_RELBRANCH)
http://hg.mozilla.org/dom-inspector/rev/bf93a1148597
http://hg.mozilla.org/chatzilla/rev/6c302afa1b50
http://hg.mozilla.org/venkman/rev/508ba977910a

L10n revisions according to opt-ins as listed in l10n-changesets (taken from the sign-off tool)

Build 2 (only changed)

http://hg.mozilla.org/comm-central/rev/402a29ffb09b (COMM20b11_20110203_RELBRANCH)

Notes

Build1

• Clobbered all release builder directories on build slaves
• Updated release-comm-central-trunk.py and symlinked release_config.py to that
  • Needed to update release_config.py to not have the key macosx due to macosx64 changes.
• Updated and reconfigured buildmaster
• Kicked off with the following command:

buildbot sendchange --username=Callek --master=localhost:9010 --branch=comm-central -c "SeaMonkey 2.1b2build1" doit

• The kick failed. We had a hung slave bug 631386 which I didn't know blocks (hangs) our buildbot reconfig. Thus my sendchanges were being ignored (buildbot thought it was sent to the wrong branch). After a few hours of chat with catlee (THANKS) and a busted buildmaster later, we figured it out, and re-sent the sendchange and got builds going.
  • Dropped that slave from our config.py for now.
  • I have a bit of debugging cruft lying around, which I'll remove within 24 hours.
• Tagging finished fine, but had a red in all other builds due to a client.py issue. Will likely change the build factory but thats a task for tomorrow for me.
• Edited seamonkey/release_master.py in buildbot-configs to have a dummy_factory for the tag builder, and then to comment out cvsroot from all the runs that use it for client.py, leaving it for the patcher runs. and reconfiged.
• re-kicked with the following command (still build 1)

buildbot sendchange --username=Callek --master=localhost:9010 --branch=comm-central -c "SeaMonkey 2.1b2build1" doit

• Forgot to clobber release dirs... so had to kick this again
• Mac repacks, uploaded to "mac64" instead of "mac" along with their [complete] updates and xpi's, I filed bug 631864 and manually moved all files to "mac" and moved on.

Signing

• We have no signing infrastructure for SeaMonkey right now, so I faked the signing step that is usually done after completion of builds and L10n repacks and before the update generation.
• Our mac builds (non en-US) uploaded to mac64/ so I had to move them over to mac/ on stage, for both updates and builds.
• Logged onto stage-old.mozilla.org and ran fakesign.sh as documented in 2.1b1 notes.

Updates and Verification

• _l10n_verify and updates started automatically, triggered by the fake-signing.
• Updates had issues on mac, since b1 was not mac64 universal binaries, had to symlink the old macosx_info.txt to macosx64_info.txt in order to let the mac update stuff complete right, backed out http://hg.mozilla.org/build/tools/rev/b0a496209027 and re-triggered.
• Following that we noticed we had issues on all platforms for updates. This was due to our switch to omnijar and listing directories without the trailing '/' in the removed-files list. Causing our updater to expect those to be _files_ not directories and failing to properly remove them. This caused bug 632682 and ewong was gracious enough to fix it. Build 2 was then spun.

Build2

• Clobbered all release builder directories on build slaves
• Updated release-comm-central-trunk.py to account for build2 and related tags (cset)
  • Reconfig failed, followup cset
  • Was sure to leave our local diff of cvsroot commenting-out in place
• Updated and reconfigured buildmaster
• Kicked off with the following command:

buildbot sendchange --username=Callek --master=localhost:9010 --branch=comm-central -c "SeaMonkey 2.1b2build2" doit

• Tagging failed due to a network/hg issue, and unfortunately during l10n tagging. Hacked up a patch to our buildbot-configs, reconfigured and kicked again with the following command:

buildbot sendchange --username=Callek --master=localhost:9010 --branch=comm-central -c "SeaMonkey 2.1b2build2" doit

Signing

• We have no signing infrastructure for SeaMonkey right now, so I faked the signing step that is usually done after completion of builds and L10n repacks and before the update generation.
• Our mac builds (non en-US) uploaded to mac64/ so I had to move them over to mac/ on stage, for both updates and builds.
• Logged onto stage-old.mozilla.org and ran fakesign.sh as documented in 2.1b1 notes.

Updates and Verification

• _l10n_verify and updates started automatically, triggered by the fake-signing.
  • L10n verification is mostly useless, since we have many new locales, and lots of expected changed strings.
  • update failed to "bump" due to wget failing ("wget of linux_info.txt failed. at MozBuild/Util.pm line 603."; of course this file did/does exist) had to re-run manually.
• update run 2 completed fine but had to run the {platform}_update_verification manually
  • the verifications were red for linux/mac, green for windows; manual testing indicates that it is/was an issue with our partials; and that extensions/* are missing in all platform partials, I removed those on stage, and hope to actually disable them on aus "soon". (them disabled on stage is not a blocker for release)

Copy Language Packs

Used langpackmove.sh as documented in 2.0b1 notes to move the langpacks into the directory we want them in for release.

Create Checksums

With make-checksums.sh as documented in 2.0.3 notes, created MD5SUMS and SHA1SUMS files containing all files we release - copying the README from last time and replacing the versions as needed, as well as doing the same for Linux x86_64.

Cleanup

Due to the update issue we found, I deleted all partials on ftp, and then went into aus2 and deleted them from staging there as well (with the following command):

rm -rf /opt/aus2/snippets/staging/*SeaMonkey-2.1b2*/*/2.1b1/*/*/*/*/partial.txt

At the same time I created /tr directories in the 2.1b2-candidates directories on ftp, so bouncer will work right.

Push To Mirrors

Used mirrorpush.sh as documented in 2.0.3 notes to finally push the files to the public dir for mirrors to pick them up.

Final Verification

Used 'Force Build' to start the final_verification builder; all tested URLs are HTTP 200 and 302 - ready for going public!

Push Updates to the beta Channel

On aus |/opt/aus2/snippets/staging/|:

~/bin/backupsnip 20110209-SeaMonkey-2.1b2
~/bin/pushsnip 20110209-SeaMonkey-2.1b2

• (There was a few 201102**-SeaMonkey-2.1b2 I ran backup and push on the latest of those)
• Log: here

Push en-US to TrendMicro for whitelisting

Uploaded the 2.1b2 Windows, en-US Setup file to trendmicro for whitelisting. Ideally this would have been uploaded a day or two ago, but I just got around to it now.

Wall Clock Time

Build

• buildbot sendchange: Thu Feb 3 17:20:52 PST 2011
  • see notes - this did not really happen then
• buildbot sendchange: Thu Feb 3 20:31:53 PST 2011
• tag:
  • start Thu Feb 3 20:39:11 2011
  • end Thu Feb 3 20:52:38 2011
  • elapsed 13 mins, 27 secs
• buildbot sendchange: Fri Feb 4 08:58:44 PST 2011
• tag (dummy): 5 sec elapsed
• buildbot sendchange (again): Fri Feb 4 09:31:45 PST 2011
• tag (dummy): 5 secs elapsed
• source:
  • start Fri Feb 4 09:35:05 2011
  • end Fri Feb 4 09:47:07 2011
  • elapsed 12 mins, 1 secs
• linux_build:
  • start Fri Feb 4 09:36:24 2011
  • end Fri Feb 4 11:53:37 2011
  • elapsed 2 hrs, 17 mins, 12 secs
• linux64_build:
  • start Fri Feb 4 10:23:01 2011
  • end Fri Feb 4 13:06:57 2011
  • elapsed 2 hrs, 43 mins, 56 secs
• win32_build:
  • start Fri Feb 4 09:35:29 2011
  • end Fri Feb 4 12:57:38 2011
  • elapsed 3 hrs, 22 mins, 9 secs
• macosx_build:
  • start Fri Feb 4 10:22:42 2011
  • end Fri Feb 4 14:06:36 2011
  • elapsed 3 hrs, 43 mins, 53 secs
• linux_repack (14 locales):
  • start Fri Feb 4 11:57:05 2011
  • end Fri Feb 4 13:10:43 2011
  • elapsed 1 hr, 13 mins, 38 secs
• win32_repack (14 locales):
  • start Fri Feb 4 13:05:52 2011
  • end Fri Feb 4 15:19:57 2011
  • elapsed 2 hrs, 14 mins, 5 secs
• macosx_repack (14 locales):
  • start Fri Feb 4 14:07:38 2011
  • end Fri Feb 4 14:36:06 2011
  • elapsed 28 mins, 28 secs

Signing

• Done manually via fakesign.sh
  • Start: Sat Feb 5 20:31:10 PST 2011
  • End: Sat Feb 5 20:31:28 PST 2011

Updates and Verification

• updates:
• linux_l10n_verification:
• win32_l10n_verification:
• macosx_l10n_verification:
• linux_update_verify:
• win32_update_verify:
• macosx_update_verify:
• final_verification:

Update Channels

• Push to mirrors:
• Push to beta: