We were down for over 2 HOURS! Not acceptable! Found out that one of our servers was stuck at a BIOS screen waiting for someone to hit "Enter". Not happy about it at all.
Are you using them as a colo? That's usually one of the first things I do when I setup a machine to be shipped off to a colo -- make sure that a remote boot doesn't require console interaction. Reduce the grub menu timeout to 5 seconds, always skip bios prompt regardless of errors, etc. Even if the colo provides the ability to remotely interact with the console, it's less hassle if you don't have to.
No, they set up the servers for us. Didn't think to ask, "please make sure the servers reboot without someone standing by to push 'enter' if needed". They told me that the server's BIOS was saying something about the RAID battery being charged and required someone to push 'enter' to continue. Thanks for the suggestion.
