Live data · Nov 6, 2026
Publisher data, every day
AI-crawler blocking, ad tech and consent on popular websites — measured by our crawler. Plus free tools for editors, PR and ad ops.
11%Blocks an AI crawler86,052 sites
22%ads.txt68,803 sites
20%Cookie consent banner68,803 sites
Global
Data
Measured by our crawler, refreshed continuously.
robots.txt
Who blocks AI crawlers
Share of popular sites that block each AI bot by name in robots.txt.
| Crawler | Operator | Global | United Kingdom | Netherlands | Italy | Spain | Japan | Latin America |
|---|---|---|---|---|---|---|---|---|
| GPTBot | OpenAI (training) | 9.8% | 51% | 48% | 8.3% | 9.6% | 3.7% | 5.9% |
| ChatGPT-User | OpenAI (user fetch) | 5.2% | 49% | 45% | 4.1% | 5.7% | 2.2% | 2.5% |
| OAI-SearchBot | OpenAI (search) | 4% | 49% | 45% | 3.5% | 4.6% | 2.1% | 1.9% |
| ClaudeBot | Anthropic | 8.7% | 51% | 47% | 7.7% | 9.1% | 3.4% | 5.7% |
| anthropic-ai | Anthropic (legacy) | 7.6% | 50% | 47% | 6.5% | 7% | 2.3% | 4.9% |
| Google-Extended | Google (Gemini) | 7.8% | 50% | 47% | 6.9% | 6.4% | 2.6% | 4.9% |
| CCBot | Common Crawl | 9.5% | 4.8% | 4.9% | 8.7% | 8.3% | 3.1% | 5.8% |
| PerplexityBot | Perplexity | 5.1% | 49% | 45% | 4.2% | 5.3% | 2.1% | 2.4% |
| Bytespider | ByteDance | 9.3% | 4.5% | 4.5% | 7.1% | 8.3% | 3.4% | 6.2% |
| Applebot-Extended | Apple | 7.7% | 3.8% | 3.4% | 5.8% | 6% | 2.7% | 4.8% |
| Amazonbot | Amazon | 8% | 51% | 48% | 7.3% | 8.5% | 3.1% | 5.6% |
| meta-externalagent | Meta | 7.9% | 51% | 47% | 7.1% | 8.2% | 3.2% | 5.1% |
| cohere-ai | Cohere | 7.3% | 50% | 46% | 6.3% | 6.4% | 2.3% | 4.8% |
| Diffbot | Diffbot | 7.2% | 50% | 46% | 6.2% | 6.4% | 2.2% | 4.9% |
Share of sites whose robots.txt blocks the bot from the homepage by name.
Global
Technologies
Market share of web technologies across 68,803 sites.
Advertising & pixels23%
- Google Marketing Platform (DV360/CM360)7.3%
- Google Publisher Tag (Ad Manager)5.9%
- Google AdSense5.6%
- Google Ads (conversion)4.7%
- OneTrust6.3%
- IAB TCF v2 API4.7%
- Cookiebot (Usercentrics)2.2%
- CookieYes1.4%
- Google Analytics 419%
- Cloudflare Web Analytics12%
- Universal Analytics (legacy)9.4%
- Yandex Metrica3.7%
- YouTube embed5.8%
- Vimeo1.9%
- Wistia0.8%
- JW Player0.6%
- Facebook SDK2.1%
- X (Twitter) embed1.2%
- AddToAny0.9%
- Instagram embed0.6%
- Awin0.3%
- Skimlinks0.2%
- impact.com0.2%
- Rakuten Advertising0.1%
14 tools
Free tools
No sign-up. Everything runs right in your browser.
SERP snippet previewPreview title and description in searchOpen Graph generatorSocial share tags with a previewads.txt validatorCheck ads.txt syntax line by lineSchema.org generatorStructured data in JSON-LDrobots.txt testerIs this path allowed for this bot?robots.txt generatorCrawler rules incl. AI botsKeyword density checkerWord and phrase frequency in a textMeta tags generatorEssential head tags in one blockUTM builderTag links with utm_* for analytics
Services
PR & ad ops services
Coverage, monitoring and ad tech audits.
Digital PRData-driven stories that earn coverage and links.Story research from dataPress materialsMedia outreach DetailsMedia monitoringMentions of your brand and topics across news sites and feeds.Source listKeyword trackingDaily digest DetailsAd operationsPixels, conversions and campaign tagging that actually match up.Pixel and conversion auditServer-side eventsUTM conventions DetailsPublisher ad tech auditads.txt, consent, tags and page weight — what costs you revenue.ads.txt and sellers.json checkConsent and TCF setupTag and bidder audit DetailsContent distributionGet your content in front of the right audiences.Channel planLink previews (OG)Syndication and newsletters DetailsAI crawler policyDecide which AI bots may use your content — and enforce it.Bot traffic reviewrobots.txt rulesllms.txt Details
Got a story or an ad stack to fix?
Write to the desk — we reply within one business day.