While we are on this topic, does anyone know the current state of art for text duplication detection algos? I understand that Google used LSH but they must have made a lot of progress since.
I'm not sure about the algorithm in use, but I what I hope is happening is that Google is now looking for the earliest publication of content when deduplicating. Most copycat sites have to copy their text from something existing, and if Google has already indexed that, they know that later versions of it are copies (and can presumably be knocked down in rank).
