The more redundant you are the more it costs. And even with redundant network routes, power distribution, backup generators there always seems to be an undiscovered SPOF left.

Murphey's Law is the Law of the Universe ;)

One hosting provider I used, I forget who, had redundant everything, except some part of the power switching to the UPSs. During a regular UPS maintenance test it failed and half their datacenter went dark.

I visited Savvis hosting facility several years ago and they had redundant generators (each about ths size of my two car garage), redundant network connections (some high-level peerage), two separate power lines from isolated parts of the grid, and at least two separate certified diesel suppliers. Oh, and VESD fire detection and grate-type floors to help with the cooling.

And I am sure a cost to match.

every reasonable co-lo has redundant power and redundant bandwidth. hell, even tiny guys like me have that.

Most co-lo power outages are not due to both incoming power feeds failing; usually it is either human error or failure in the power equipment that is not redundant enough. It happens even at the best data centers.

Most network outages, on the other hand, are caused by human error. it's not very difficult to make your network extremely resilient to upstream failures; Even the smallest ISP is going to have more than one upstream. however, if you give the new guy access to the BGP routers, (or the old guy, when he hasn't had enough sleep) it's not at all difficult to break the whole thing.

well if its fully redundant, it'd be 2x as much.

Personally I think, the best bet is to just have a backup on Amazon. Sure its not as fast as switching to another dedicated server, but its just something to have in case of emergency, users can afford to have the page take an extra ms to load.

The cost to users (and especially developers) most assuredly is not simply 2x that of a single host. Planning for full replication and hot-failover from day one adds complexity, and can actually do more harm than good if your team and tools aren't fully up-to-speed on the sometimes subtle challenges surrounding replication lag, eventually-consistent stores, and STONITH.

My thinking was, if they already own a datacenter in another location, why not add another floor? Obviously it's not cheap but is it really 2x? If you're big enough, you don't have to hire double the staff or buy double the resources since you already have them.

The expensive part about data centers isn't the square footage, it's the HVAC/power/bandwidth/servers. So you have the space, that means you still need to reserve twice the bandwidth, power, cooling and hardware. x2 cost. Actually somewhat more because you need to account for all the bandwidth and time to keep everything synced up.

at what rackspace can charge, new hardware is nothing. the problem is that if I rent you a server, it's going to be pretty difficult for me to replicate that server to another location without a lot of cooperation from you. the more I screw with your server, well, the more likely I am to break it, too.

It's more than that as you two datacenters, plus the cost of synchronizing it.

for shared hosting, yes, there is no excuse for not being redundant. However, if you are renting servers, full redundancy at the server level is quite difficult to achieve. Redundancy is usually better handled at the application layer.