12%
Blocks an AI crawler
10,566 / 86,147
11%
Blocks GPTBot
9,809 / 86,147
11%
Blocks CCBot
9,713 / 86,147
9.7%
Blocks Google-Extended
8,381 / 86,147
AI crawlers blocked in robots.txt
| Crawler | Operator | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|---|
| GPTBot | OpenAI (training) | 11% | 53% | 46% | 9.7% | 10% | 4.4% | 8.4% |
| ChatGPT-User | OpenAI (user fetch) | 4.9% | 49% | 43% | 3.9% | 5.3% | 2.1% | 2.4% |
| OAI-SearchBot | OpenAI (search) | 3.6% | 49% | 42% | 3.4% | 4.5% | 2% | 1.8% |
| ClaudeBot | Anthropic | 11% | 53% | 46% | 9.4% | 9.6% | 4% | 8.2% |
| anthropic-ai | Anthropic (legacy) | 9.6% | 52% | 45% | 8.2% | 8% | 3% | 7.4% |
| Google-Extended | Google (Gemini) | 9.7% | 52% | 45% | 8.6% | 7.5% | 3.3% | 7.5% |
| CCBot | Common Crawl | 11% | 5.8% | 5.4% | 9.6% | 8.5% | 3.7% | 8.3% |
| PerplexityBot | Perplexity | 4.7% | 49% | 42% | 4% | 4.9% | 2.1% | 2.2% |
| Bytespider | ByteDance | 11% | 5.5% | 5.1% | 8.4% | 8.4% | 4% | 8.7% |
| Applebot-Extended | Apple | 9.6% | 4.9% | 4.1% | 7.3% | 6.4% | 3.4% | 7.4% |
| Amazonbot | Amazon | 9.9% | 52% | 46% | 9% | 9.3% | 3.6% | 8.1% |
| meta-externalagent | Meta | 9.8% | 52% | 45% | 8.9% | 9.1% | 3.9% | 7.7% |
| cohere-ai | Cohere | 9.3% | 52% | 45% | 8% | 7.6% | 3% | 7.4% |
| Diffbot | Diffbot | 9.2% | 52% | 45% | 8% | 7.7% | 2.8% | 7.5% |
Share of sites whose robots.txt blocks the bot from the homepage by name.
By market
| Metric | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|
| Blocks an AI crawler | 12% | 6.5% | 6.7% | 11% | 11% | 4.5% | 8.9% |
| Blocks GPTBot | 11% | 53% | 46% | 9.7% | 10% | 4.4% | 8.4% |
| Blocks ClaudeBot | 11% | 53% | 46% | 9.4% | 9.6% | 4% | 8.2% |
| Blocks CCBot | 11% | 5.8% | 5.4% | 9.6% | 8.5% | 3.7% | 8.3% |
| Blocks Google-Extended | 9.7% | 52% | 45% | 8.6% | 7.5% | 3.3% | 7.5% |
| Blocks PerplexityBot | 4.7% | 49% | 42% | 4% | 4.9% | 2.1% | 2.2% |
| Content-Signal in robots.txt | 6.8% | 3.7% | 3.1% | 5.5% | 4.2% | 1.2% | 6.7% |
| llms.txt | 15% | 13% | 11% | 12% | 13% | 5.9% | 18% |
Search bots blocked
- 1PetalBot15%8,115
- 2Baiduspider13%7,160
- 3YandexBot5.9%3,200
- 4Applebot4.6%2,502
- 5SeznamBot4.5%2,410
- 6Sogou4.1%2,212
- 7Naverbot4.1%2,190
- 8Yeti4.1%2,187
- 9DuckDuckBot3.3%1,758
- 10Bingbot3.1%1,671
Share of 53,980 sites in the sample.
SEO bots blocked
- 1MJ12bot8.7%4,699
- 2AhrefsBot8.4%4,508
- 3SemrushBot8%4,322
- 4DotBot7.9%4,248
- 5BLEXBot7%3,776
- 6DataForSeoBot6.5%3,521
- 7Barkrowler5.1%2,771
- 8rogerbot4.6%2,501
- 9serpstatbot4.3%2,348
- 10Screaming Frog SEO Spider4%2,150
Share of 53,980 sites in the sample.
Our crawler: 68,167 sites · Global · updated Nov 13, 2026 · Methodology
Got a story or an ad stack to fix?
Write to the desk — we reply within one business day.