Release Engineering Buildduty Meeting

• Date: 2013-10-22
• Time: 10:00am EDT
• Room: ReleaseEngineering Vidyo room
• Meeting notes: https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Meetings:2013-10-22

Status of buildduty period

https://releng.etherpad.mozilla.org/buildduty

Bugs filed

Bug 925772 - Log tegra error.flg failures somewhere persistent

Previous action items

• bhearsum to add documentation at https://wiki.mozilla.org/ReleaseEngineering/Applications/SlaveAPI
• (carry over) armenzg
  • propose a query for non-problem tracking VS buildduty bugs
• coop:
  • (carry over) update nagios buildduty docs
  • (carry over) follow-up with IT re: JSON formatting of nagios output
  • add existing buildduty queries to wiki
  • talk to arr about turning off nagios slave checks in IRC
    • https://bugzilla.mozilla.org/show_bug.cgi?id=927941
  • file bug for manual smoketest of cedar build
    • https://bugzilla.mozilla.org/show_bug.cgi?id=927951
  • file bug to audit releng services for IP collisions

Agenda

• (bhearsum) - time to finalize slaveapi reboot bug updating
  • need to be visible somewhere
  • kittenherder replacement needs to look at combination of status (eg, reboot history, recent jobs, etc)
  • log to bugzilla until we have persistent storage
  • file some sort of "poke me" bug instead of specific "reboot"
  • put last job information in bugzilla message
  • maybe pull bug escalation out of reboot, let client make decision
• (coop) buildduty coverage
  • swap 2 meeting-heavy days with someone else?
  • coop does 1 day/week?
• (coop) how to empower others to make decisions about slaves
  • 3 strikes
  • how to avoid partially re-imaged slaves from burning multiple jobs
  • hook on failed jobs to check slave history, disable slave after 5 failures?
• (jhopkins) win64-rev2 update
• (jhopkins) win64-rev1 post-imaging

List of current projects

• https://github.com/bhearsum/slaveapi/blob/master/TODO
• https://bugzilla.mozilla.org/show_bug.cgi?id=914764
• in-house capacity: https://bugzilla.mozilla.org/show_bug.cgi?id=867593

Action items

• armenzg:
  • propose a query for non-problem tracking VS buildduty bugs
• coop:
  • update nagios buildduty docs carryover
  • follow-up with IT re: JSON formatting of nagios output
  • add existing buildduty queries to wiki
  • file bug to audit releng services for IP collision
• jhopkins: talk to buildduty about whether to take bugs

930021 and 930216 into the buildduty queue

• • should we have a separate monitoring component created?
  • "other" component has many bugs in it; bugs can become forgotten