:: ~/website ‹master*› » rg '(GPTBot|ClaudeBot|Bytespider|Amazonbot)' access.log | awk '{print $1}' | sort -u | wc -l
15163
map $http_user_agent $uatype { default 'user'; ~*(googlebot|bingbot) 'good_bot'; ~*(nastybot|somebadscraper) 'bad_bot'; }
I’m not dismissing the difficulty of the problem but there are multiple vectors that can identify these ‘bad_bots’.
:: ~/website ‹master*› » rg '(GPTBot|ClaudeBot|Bytespider|Amazonbot)' access.log | awk '{print $1}' | sort -u | wc -l
15163