Per-crawler totals, 746 sites with a readable robots.txt, checked 21 to 22 October 2026 (number of sites that block it, and what the operator says it's for):
CCBot (Common Crawl, training): 146
Bytespider (ByteDance, training): 137
ClaudeBot (Anthropic, training): 135
Applebot-Extended (Apple, token for AI use of Applebot's fetches): 127
GPTBot (OpenAI, training): 126
Meta-ExternalAgent (Meta, training): 125
PerplexityBot (Perplexity, search): 114
Google-Extended (Google, token for AI use of Googlebot's fetches): 101
Amazonbot (Amazon, general): 99
OAI-SearchBot (OpenAI, search): 70
ChatGPT-User (OpenAI, fetches triggered by a user): 69
Claude-User (Anthropic, fetches triggered by a user): 66
Perplexity-User (Perplexity, fetches triggered by a user): 65
Claude-SearchBot (Anthropic, search): 62
Same data as a CSV, one row per site: getkavel.com/ai-crawlers/data.csv
Per-crawler totals, 746 sites with a readable robots.txt, checked 21 to 22 October 2026 (number of sites that block it, and what the operator says it's for):
CCBot (Common Crawl, training): 146 Bytespider (ByteDance, training): 137 ClaudeBot (Anthropic, training): 135 Applebot-Extended (Apple, token for AI use of Applebot's fetches): 127 GPTBot (OpenAI, training): 126 Meta-ExternalAgent (Meta, training): 125 PerplexityBot (Perplexity, search): 114 Google-Extended (Google, token for AI use of Googlebot's fetches): 101 Amazonbot (Amazon, general): 99 OAI-SearchBot (OpenAI, search): 70 ChatGPT-User (OpenAI, fetches triggered by a user): 69 Claude-User (Anthropic, fetches triggered by a user): 66 Perplexity-User (Perplexity, fetches triggered by a user): 65 Claude-SearchBot (Anthropic, search): 62
Same data as a CSV, one row per site: getkavel.com/ai-crawlers/data.csv
[flagged]
[flagged]