Skip to content
Live data
Blocks an AI crawler 12%Blocks GPTBot 11%ads.txt 22%Cookie consent banner 20%Ad pixels 22%llms.txt 15%HTTPS homepage 99%DMARC record 65%Blocks an AI crawler 12%Blocks GPTBot 11%ads.txt 22%Cookie consent banner 20%Ad pixels 22%llms.txt 15%HTTPS homepage 99%DMARC record 65%
TNP.media
Live data · Nov 20, 2026

Publisher data, every day

AI-crawler blocking, ad tech and consent on popular websites — measured by our crawler. Plus free tools for editors, PR and ad ops.

12%Blocks an AI crawler86,066 sites
22%ads.txt68,085 sites
20%Cookie consent banner68,085 sites
robots.txt

Who blocks AI crawlers

Share of popular sites that block each AI bot by name in robots.txt.

CrawlerOperatorGlobalUnited KingdomNetherlandsItalySpainJapanLatin America
GPTBotOpenAI (training)11%54%46%10%10%4.3%8.4%
ChatGPT-UserOpenAI (user fetch)4.9%51%43%4.8%5.7%2.1%2.5%
OAI-SearchBotOpenAI (search)3.6%50%42%4.3%4.9%2%1.9%
ClaudeBotAnthropic10%54%46%10%9.7%4%8.2%
anthropic-aiAnthropic (legacy)9.3%53%45%8.9%8.1%3%7.4%
Google-ExtendedGoogle (Gemini)9.5%53%45%9.3%7.6%3.3%7.5%
CCBotCommon Crawl11%5.4%5.4%9.4%7.9%3.7%8.2%
PerplexityBotPerplexity4.7%50%43%4.9%5.4%2.1%2.3%
BytespiderByteDance11%5.2%5.2%8.2%7.9%4%8.6%
Applebot-ExtendedApple9.4%4.6%4.1%7%5.9%3.3%7.3%
AmazonbotAmazon9.7%54%46%9.8%9.3%3.6%8.1%
meta-externalagentMeta9.6%54%46%9.6%9.2%3.9%7.7%
cohere-aiCohere9.1%53%45%8.8%7.8%3%7.4%
DiffbotDiffbot9%53%45%8.8%7.9%2.8%7.5%

Share of sites whose robots.txt blocks the bot from the homepage by name.

Got a story or an ad stack to fix?

Write to the desk — we reply within one business day.