In addition to crawl-delay there's also HTTP 429 and the retry-after header. htt... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		the8472 on April 4, 2020 \| parent \| context \| favorite \| on: Ask HN: Best practices for ethical web scraping? In addition to crawl-delay there's also HTTP 429 and the retry-after header. https://tools.ietf.org/html/rfc6585#page-3

wumpus on April 4, 2020 [–]

Sites also use 403 and 503 to send rate-limit signals, despite what the RFCs say.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact