Thunderbird-45.0/BuildNotes

From MozillaWiki
Jump to: navigation, search
  • win32 build step failed for Thunderbird 45.0 build1, bash crashed during the generation of a partial update, which is an intermittent issue. Rerunning the job. nthomas reran
  • release-comm-esr45-win32_repack_8/10 hit a crash in bash while creating an update, reran.
  • release-comm-esr45-win32_repack_8/10 and 5/10 hit a hang while creating an update, reran both.
  • lots or retries on windows repacks, as slaves disconnected from masters
  • repack_5/10 failed for Thunderbird 45.0 build1 on win32 failed many times with bash error, rerunning now
  • release-comm-esr45-thunderbird_antivirus failed connecting to s3. Maybe we should add a retry there? Will have to go look at code.
9:51:51     INFO - retry: Calling <bound method Key.get_contents_to_filename of <Key: net-mozaws-prod-delivery-archive,pub/thunderbird/candidates/45.0-candidates/build1/update/linux-x86_64/ro/thunderbird-45.0.complete.mar>> with args: (u'cache/update/linux-x86_64/ro/thunderbird-45.0.complete.mar',), kwargs: {}, attempt #5
09:51:51     INFO - retry: Giving up on <bound method Key.get_contents_to_filename of <Key: net-mozaws-prod-delivery-archive,pub/thunderbird/candidates/45.0-candidates/build1/update/linux-x86_64/ro/thunderbird-45.0.complete.mar>>
09:51:51    FATAL - Uncaught exception: Traceback (most recent call last):
09:51:51    FATAL -   File "/builds/slave/tb-rel-c-esr45-av-000000000000/scripts/mozharness/base/script.py", line 1756, in run
09:51:51    FATAL -     self.run_action(action)
09:51:51    FATAL -   File "/builds/slave/tb-rel-c-esr45-av-000000000000/scripts/mozharness/base/script.py", line 1698, in run_action
09:51:51    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
09:51:51    FATAL -   File "/builds/slave/tb-rel-c-esr45-av-000000000000/scripts/mozharness/base/script.py", line 1639, in _possibly_run_method
09:51:51    FATAL -     return getattr(self, method_name)()
09:51:51    FATAL -   File "scripts/scripts/release/antivirus.py", line 177, in get_files
09:51:51    FATAL -     pool.map(worker, find_release_files())
09:51:51    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 227, in map
09:51:51    FATAL -     return self.map_async(func, iterable, chunksize).get()
09:51:51    FATAL -   File "/tools/python27/lib/python2.7/multiprocessing/pool.py", line 528, in get
09:51:51    FATAL -     raise self._value
09:51:51    FATAL - IncompleteRead: IncompleteRead(0 bytes read, 32737227 more expected)

Second run of this job succeeded.

  • release updates failed because patcher config was missing, hg cp mozEsr38-thunderbird-branch-patcher2.cfg mozEsr45-thunderbird-branch-patcher2.cfg in tools and reran job. Then it failed submitting to balrog because of missing rules. Fixed in bug 1263279 When I ran the job again it failed since the it couldn't commit configs with the same tag, will have to run balrog job manually and retrigger downstream jobs.
  • I changed the cdntest rule (id 334) to remove the (Version: >45.0) restriction and set the rate to 10 as per bug bug 1263279
  • ran checksums and antivirus builders by hand since upstream jobs had failed
  • final verification failed. When the version bump part of the release updates job ran

http://buildbot-master74.bb.releng.usw2.mozilla.com:8001/builders/release-comm-esr45-thunderbird_release_updates/builds/4

It didn't update the update configs, only the patcher configs.

changeset: 6700:64c64b628a31 tag: THUNDERBIRD_45_0_BUILD1_RUNTIME tag: THUNDERBIRD_45_0_RELEASE_RUNTIME user: tbirdbld date: Mon Apr 11 15:09:18 2016 -0400 files: release/patcher-configs/mozEsr45-thunderbird-branch-patcher2.cfg description: Automated configuration bump: update configs for Thunderbird 45.0 build 1 on release 

4:55 PM <rail> python tools/scripts/updates/create-update-verify-configs.py -c tools/release/patcher-configs/mozEsr45-thunderbird-branch-patcher2.cfg --platform macosx64 --output tools/release/updates/mozEsr45-thunderbird-mac64.cfg --release-config-file mozilla/release-thunderbird-comm-esr45.py -b https://hg.mozilla.org/build/buildbot-configs --channel release-localtest -t
4:55 PM <rail> THUNDERBIRD_45_0_RELEASE
4:55 PM <rail> with tools and buildbot-config cloned in the $PWD
4:55 PM <kmoir> okay
4:56 PM <rail> repeat it for linux/linux64/win32 (need to change the cmd in 2 places) and you are done
4:56 PM <rail> you will need to commit and tag it with the something_RUNTIME tag

I did this and reran the final_release_verification job. It failed because the update tests don't deal with the 38.0 and 45.0 split. I changed the balrog rules so the TB team could test updates on release-cdntest (from 38.x to 45.x) (removed version restriction and increased priority). Once they conclude their testing, we can set the balrog rule for release for 45.0 to remove the version restriction temporarily and set a low rate (10). (Rate was set to 10 on release channel on Friday morning, will set it to 0 on Friday night)

  • Friday at 5pm ET - set the rate back to 0 and Version to > 45.0 for the release channel for tb 45.0
  • Tuesday Apr 19 at 5pm ET - * allow updates from version 38 to 45, but keep background rate at 0
  • Wed Apr 20 at 4pm ET - set background rate to 10
  • Mon Apr 25 at 8:30 ET - set background rate to 50
Retrieved from "https://wiki.mozilla.org/index.php?title=Thunderbird-45.0/BuildNotes&oldid=1129150"