ReleaseEngineering/BuildFarmRefresh/2012Q2

From MozillaWiki
Jump to: navigation, search

Platform priority

The following platform priority list is from a recent Channel meeting, but has been extended to include existing platforms and new platforms that are already in-flight:

  • OS X 10.8 signing of 10.7 builds
  • unified Windows builders (single 64bit image that can create Win8, win64, win32-on-win64 builds to prevent linker issues in the future)
  • Windows 8 build hardware
  • Windows 8 test hardware
  • b2g
  • unified Linux builders (single 64bit image+mock)
  • Rapid Betas
  • mobile test hardware (pandaboards)
  • builds for armv6
  • builds for android-on-intel
  • Mac 10.8 test hardware
  • Mac 10.8 build hardware?
  • new Windows test hardware
  • new Linux test hardware

Major goals

  • tackle many targets in parallel rather than serially as in the past
    • requires investing more than one human resource
  • virtualize as much hardware as is possible
    • caveats:
      • only works for building and unittests(correctness), not talos (performance)
      • build times must be competitive with hardware:
        • perf: 25% slower may be acceptable, order-of-magnitude is not
        • cost: costs of bulk AWS VMs? costs of VMs (incl ESX lic.), netapps, hosts?
      • should VMs be able to run on both AWS and Mozilla hardware?
  • Queue collapsing
    • currently using queue collapsing to mitigate lack of capacity.
    • should have enough capacity that queue collapsing is again considered an error condition; setup nagios alerts when queue collapsing triggered.

Plan of attack

  1. acquire signing key from Apple
    • releng: joduinn, catlee
  2. setup 10.8 signing machine
  3. segregate 5 Win64 build machines to be exclusively used as Win8 builders until a unified image is available
  4. create single linux64 VM that can build all current linux-based builds: linux32, linux64, android*, b2g
    • mock envs for each build type
      • catlee already has an image on AWS that does almost all of this AIUI. Still many services issues to be worked out there though.
    • based on something more recent than CentOS 5 (CentOS 6?)
    • virtualizing b2g builders frees up DL120 hardware for use as Win8 testers
    • virtualizing all linux builders frees up ix machines for use as Windows builders
    • releng: catlee, bear; IT: arr
  5. create single Win64 hardware image that can build all current Windows-based builds: Win32, Win64, Win8
  6. setup DL120s as Win8 testers
    • buy graphics cards for existing DL120s (have 3, need 50)
    • determine how many more systems we need to purchase
  7. rackmount enclosure for pandaboards
    • still blocked on initial automation work by A-Team
  8. create mock env for armv6
  9. create mock env for android-on-intel

Open questions

  • switching (back!) to VMs for unittests
    • switching unittests from VMs to hardware reduce intermittent oranges.
    • is this practical?
  • where should VM capacity come from (AWS vs Mozilla IT vs ???)
  • capacity for existing platforms. Do we need more:
    • tegras?
    • rev5 builders?
    • rev5 testers?
  • new hardware for general Linux and Windows testing?
    • same hardware between platforms? (no requirement that it be the same from the dev side)
    • DL120s?
    • how many, and for which OSes?