The level of discourse on HN lately is absurd. Here is a new CEO that appears to be looking to change things for the positive, yet everyone wants to be cynical and doubting. Yet when startups have major f* ups, the conversation is much different.

Lets all admit change is slow and that paypal is still one of the more trusted payment solutions accepted globally. It would be nice to be able to use them in the future with some continued reliability.

There's something ironic about combination of "Mike Arrington" and "turning the drama down".

During my tenure at NetApp I got to see all sorts of really really interesting disk problems and the lengths software had to go to reliably store data on them. Two scars burned into me from that were 1) disks suck, and 2) disks are not 'storage'.

The first part is pretty easy to understand, storage manufacturers have competed for years in a commodity market where consumers often choose price per gigabyte over URER (the unrecoverable read error rate), further at the scale of the market small savings of cents adds up to better margins. And while the 'enterprise' fibre channel and SCSI drives could (and did) command a hefty premium, the shift to SATA drives really put a crimp in the over all margin picture. So the disk manufacturers are stuck between the reliability of the drive and the cost of the drive. They surf the curve where it is just reliable enough to keep people buying drives.

This trend bites back, making the likely hood of an error while reading more and more probable. Not picking on Seagate here, they are all similar, but lets look at their Barracuda drive's spec sheet [1]. You will notice a parameter 'Nonrecoverable Read Errors per Bits read', and you'll see that its 1x10e14 across the sheet, from 3TB down to 250GB. It is a statistical thing, the whole magnetic field domain to digital bit pipeline is one giant analog loop of a error extraction. 1x10E14 bits. So what does that mean? Lets say each byte is encoded in 10 bits on the disk. Three trillion bytes is 30 trillion bits in that case, or 3x10E13 bits. Best case, if you read that disk from one end to the other (like you were re-silvering a mirror from a RAID 10 setup) you have a 1 in 3 chance that one of those sectors won't be readable for a perfectly working disk. Amazing isn't it? Trying to reconstruct a RAID5 group with 4 disks remaining out of 5, look out.

So physics is not on your side, but we've got RAID 6 Chuck! Of course you do, and that is a good thing. But what about when you write to the disk, the disk replies "Yeah sure boss! Got that on the platters now" but it didn't actually right it? Now you've got a parity failure waiting to bite you sitting on the drive, or worse, it does write the sector but writes it somewhere else (saw this a number of times at NetApp as well). There are three million lines of code in the disk firmware. One of the manufacturers (my memory says Maxtor) showed a demo where they booted Linux on the controller of the drive.

Bottom line is that the system works mostly, which is a huge win, and a lot of people blame the OS when the drive is at fault, another bonus for manufacturers, but unless your data is in a RAID 6 group or on at least 3 drives, its not really 'safe' in the "I am absolutely positive I could read this back" sense of the word.

[1] http://www.seagate.com/files/staticfiles/docs/pdf/datasheet/...