ReleaseEngineering/BuildFarmRefresh/2012Q2
From MozillaWiki
Platform priority
The following platform priority list is from a recent Channel meeting, but has been extended to include existing platforms and new platforms that are already in-flight:
- OS X 10.8 signing of 10.7 builds
- unified Windows builders (single 64bit image that can create Win8, win64, win32-on-win64 builds to prevent linker issues in the future)
- Windows 8 build hardware
- Windows 8 test hardware
- b2g
- unified Linux builders (single 64bit image+mock)
- Rapid Betas
- mobile test hardware (pandaboards)
- builds for armv6
- builds for android-on-intel
- Mac 10.8 test hardware
- Mac 10.8 build hardware?
- new Windows test hardware
- new Linux test hardware
Major goals
- tackle many targets in parallel rather than serially as in the past
- requires investing more than one human resource
- virtualize as much hardware as is possible
- caveats:
- only works for building and unittests(correctness), not talos (performance)
- build times must be competitive with hardware:
- perf: 25% slower may be acceptable, order-of-magnitude is not
- cost: costs of bulk AWS VMs? costs of VMs (incl ESX lic.), netapps, hosts?
- should VMs be able to run on both AWS and Mozilla hardware?
- caveats:
- Queue collapsing
- currently using queue collapsing to mitigate lack of capacity.
- should have enough capacity that queue collapsing is again considered an error condition; setup nagios alerts when queue collapsing triggered.
Plan of attack
- acquire signing key from Apple
- releng: joduinn, catlee
- setup 10.8 signing machine
- https://bugzilla.mozilla.org/show_bug.cgi?id=744789
- releng: bhearsum
- segregate 5 Win64 build machines to be exclusively used as Win8 builders until a unified image is available
- unblocks developers immediately
- tools can be installed by hand until GPO is ready
- releng: armenzg; IT: MaRu
- https://bugzilla.mozilla.org/show_bug.cgi?id=754008
- https://bugzilla.mozilla.org/show_bug.cgi?id=737994
- create single linux64 VM that can build all current linux-based builds: linux32, linux64, android*, b2g
- mock envs for each build type
- catlee already has an image on AWS that does almost all of this AIUI. Still many services issues to be worked out there though.
- based on something more recent than CentOS 5 (CentOS 6?)
- virtualizing b2g builders frees up DL120 hardware for use as Win8 testers
- virtualizing all linux builders frees up ix machines for use as Windows builders
- releng: catlee, bear; IT: arr
- mock envs for each build type
- create single Win64 hardware image that can build all current Windows-based builds: Win32, Win64, Win8
- image all ix machines with new Win64 image
- releng: catlee, armenzg; IT: MaRu
- https://bugzilla.mozilla.org/show_bug.cgi?id=753132
- setup DL120s as Win8 testers
- buy graphics cards for existing DL120s (have 3, need 50)
- determine how many more systems we need to purchase
- rackmount enclosure for pandaboards
- still blocked on initial automation work by A-Team
- create mock env for armv6
- create mock env for android-on-intel
Open questions
- switching (back!) to VMs for unittests
- switching unittests from VMs to hardware reduce intermittent oranges.
- is this practical?
- where should VM capacity come from (AWS vs Mozilla IT vs ???)
- capacity for existing platforms. Do we need more:
- tegras?
- rev5 builders?
- rev5 testers?
- new hardware for general Linux and Windows testing?
- same hardware between platforms? (no requirement that it be the same from the dev side)
- DL120s?
- how many, and for which OSes?