Often we need to troubleshoot the hardware workers for various reasons:

• falling off the network
• generic worker/OCC fail to start or not running
• hardware failure
• not picking up tasks

If you notice any releng-hardware workers missing or not picking tasks, escalate to ciduty in #ci.

Windows 10

When a windows machine needs to be action-ed upon the best place to start is its logs. IF the logs aren't showing the worker ready for tasks reboot it from the worker explorer. Sometimes a reboot won't do the trick and in this case machine needs to be re-imaged. Following the moonshot spreadsheet, re-image the machine through the HP iLO Integrated Remote Console. Be sure to follow the process until it completes and check back on it in the worker explorer to see if it's picking tasks again.

Linux 64

When a linux machine needs to be action-ed upon the best place to start is its logs. If a linux worker stopped picking tasks, reboot it from the worker explorer. Machines usually recover from this, if not re-image them following the moonshot spreadsheet and these steps. You should be getting a puppet e-mail regarding the re-imaged worker. Remember to check back on it in the worker explorer to see if it's picking tasks again.

OSX 10.10

When OSX machines stop taking tasks reboot them from the worker explorer. Most of the time this recovers the worker. Otherwise re-image it by running the appropriate line as per this.