Why not just lower the crawl rate? My search engine crawler visits tens of million documents in a week at a rate of 1 doc/second, but a few hundred different domains at the same time.

Going as low as 0.2 dps could easily be doable I think.