Buildbot/OutageReports/20070905-01

2007-09-05

On 2007-09-05 at 21:30PDT, WINNT 5.1 qm-pxp01,3 talos trunk experienced a service outage for 3 hours. Tree was closed around 21:30PDT and remained closed for 2 more hours.

bug 394982

What was affected:

Performance reporting on trunk.

What was the cause of the outage:

Update to configuration files was not propagated internally via a reconfig request.

Has this type of outage happened before?

Yes - I've seen this happen before. Changes to config files were not picked up on reconfig.

What was done to repair:

Build master was restarted.

What will be done to prevent this in the future:

either upgrade buildbot to trunk code or not rely on reconfigs for important changes.