To build on this, we distrusted the actual servers in the rack for the following:
1. Out of band devices to administer boot fail incidents, serially controlled
2. PDU (Power distribution Unit). Supplied with 2 different power sources. Provides dual power per server on rack. Controlled by ethernet.
3. Firewall rules that prevent the managed crap (iLO and the like) from working.
The only recent issue my previous job had, was in a junior sysad installed apache tomcat with default creds. Can't really fix user error - but the above rules at least stop malicious intent from manufacturers. Mostly.
Thanks, HP! /s