When I worked IT for a medium sized university we were asked by the DOJ to install switches that copied all internet traffic directly to an unspecified government server. We were told all ISPs (anyone providing internet to more than 100 persons) were told to do this as well.
We refused to comply obviously as the request was absurd, but in the small print of the request we were told we were not allowed to speak of the request and were to deny any involvement if asked under some unknown penalty.
I wouldn't be surprised if special terms of Google's interaction with any government agencies has a similar clause.
The NSA's Utah data center was designed to host more storage than existed on the entire planet at the time. Sheer volume of data is among the least of their concerns.
OK, so you can read wikipedia, but does it make any sense? A single 65MW datacenter with only 100ksqft of machine room space can hold all of the data in the world? That's not even as much power as one of Facebook's facilities in Oregon.
Remember when IBM announced they'd built the largest filesystem in history? That was only 120PB. A YB is 10 million times bigger than that.
Consider also that a YB is 1000x the entire storage industry output from 2012. So under your theory there is a parallel hard disk drive industry consisting of at least 99% of all hard disk production on Earth.
>" A single 65MW datacenter with only 100ksqft of machine room space can hold all of the data in the world? That's not even as much power as one of Facebook's facilities in Oregon."
You're off by a factor of 10-15 on the square footage. Per the wiki page, "The planned structure is 1 million or 1.5 million square feet."
"According to USACE, the center will have 100,000 square feet of raised-floor data center space and more than 900,000 square feet of technical support and administrative space."
Facebook's servers spend most of their time finding, sorting and then transmitting bits to the billion people who use Facebook. Storing a bunch of data and storing a bunch of data that needs to be dynamically assembled in real time for a billion users are two very different things.
Regardless, no one has theorized that the NSA storing yottabytes of data. The (still probably crazy) rumors are zettabytes.
The NSA wouldn't be storing all the photos or videos from Facebook. Storing text is trivial compared to that.
If you reduced Facebook's storage demands down to always static text, I think there's little doubt you could eliminate over 90% of their storage requirements (I'd say closer to 97%). And that's without any kind of compression.
According to the Wired story on it, it's four 25,000 square foot data center buildings[1]. That's smaller than one of Apple's smaller data centers. Apple's Oregon one is going to be 500,000 square feet.
It's not going to be able to hold more than an average size datacenter, and there are a lot of data centers out there, each holding more data that would have to be copied. The NSA has huge capabilities, but they're not nearly there yet. Most of what we should be worried about is their legal capabilities.
edit: it's also worth noting that the wikipedia article's source is the wired article, and all the wired article says is that a DoD report says that they need to start designing their network to handle yottabytes, which isn't even related to the NSA's data center. Wired then does some interpolation by using "yottabyte" to make the requirements of the data center sound big, but they never say that it was designed to handle that level of data or that it will ever be able to handle that level of data (in fact, they never even cite any level of data that it was designed to handle). But the Wikipedia article cites the Wired article, and suddenly it's become "a data storage facility for the United States Intelligence Community that is designed to store data on the scale of yottabytes".
> It's not going to be able to hold more than an average size datacenter, and there are a lot of data centers out there. T
Why not? Most data centers aren't designed to store a bunch of stuff, but instead are designed to serve a lot of users. Facebook has at least one storage forward data center and it packs an exabyte into a 60k sqft footprint:
I was replying to the fantastical claim that the datacenter "was designed to host more storage than existed on the entire planet at the time". Of course 100,000 square feet is a medium sized data center and can hold quite a bit of data. If you're happy to go even slower, you could fit even more in with a tape archive that big. You're still so very very far away from a data center sucking up all human-produced data (even accounting for duplication) and storing it for later retrieval and analysis as many here apparently believe.
So yes, the NSA does have a new datacenter. But no, it's not the new Area 51 nor is it host to magic computers.
I'd like to point out that their upper capacity limit, "a trillion terabytes", would require approximately 1 trillion hard drives, probably at a cost of $10-$100 trillion. I feel comfortable saying that they have not built out to that scale yet.
They don't need to access to all network traffic, just the interesting bits, like e.g. the internal traffic for the storage/backup systems used for GMail or Google Accounts .
Word. I get that. I just thought the question sort of steered away from the main point of the parent comment, which was that if Google did have some sort of nefarious partnership that they would be legally obliged to deny it no matter what. My apologies, I'm probably just being a bit nitpicky.
PRISM is clearly not the whole picture. The Utah Data Center alone is much more than $20M a year and PRISM's specified purpose was using US IT infrastructure to intercept foreign communications, which would presumably be filtered from a large data set.
The bulk of the data transmission and storage costs are probably under another budget
The program and requirements were somewhat surreal. The architecture of the plan seemed to make little to no sense.
They considered anyone who provided internet access to 100's of people a provider (mostly universities etc) and requested that they install the pre specified hardware at their own expense (purchase, installation and maintenance).
They may have been concerned specifically with universities due to their foreign students and access to certain backbone lines that normal ISP's dont always have (WWW2?).
I spoke with the EFF and I was told I wasn't alone, but most people were rejecting the 'request' and not suffering any consequences. Our IT director took a hard line on the topic and tried to speak openly to reporters about how much the request offended him. He got as far as the school newspaper as in 2005 most people didn't understand the ramifications of this activity.
Presumably university IT would have been able to provide more precise demographically identifying data than the ISP. It's unlikely the ISP was running the RADIUS server.
You're right, I hadn't considered that. Identifying individual students within public areas of the network was only possible through the schools ID system. Otherwise public access would be anyones guess as to who was using the computer.
You're not allowed to talk about NSLs, i recommend you change your post. Whats really bad about NSLs is they last forever, even a decade after receiving one you still can't talk about it without risking prosecution. NSLs issued in 2002 have no bearings on any active case yet we can't talk about them publicly. This makes it really hard to have a public debate on their usage.
We refused to comply obviously as the request was absurd, but in the small print of the request we were told we were not allowed to speak of the request and were to deny any involvement if asked under some unknown penalty.
I wouldn't be surprised if special terms of Google's interaction with any government agencies has a similar clause.