You can easily protect your hardware from all but the most determined adversaries with extensive physical access. Epoxy in ports, case intrusion detection and locked down boot chain. Use TPM2-totp for verified boot.

Your colo provider can be thoroughly owned, your adversaries can have physical access to the server for extended periods of time and still not be able to do anything because you've denied them access to any ports that'd allow DMA.

Lots of cheap DIY options for fancy case intrusion detection going way beyond that offered by mfgs. USB camera and some tape?