Skip to content
Live data
Blocks an AI crawler 10%Blocks GPTBot 9.6%ads.txt 22%Cookie consent banner 19%Ad pixels 23%llms.txt 15%HTTPS homepage 98%DMARC record 64%Blocks an AI crawler 10%Blocks GPTBot 9.6%ads.txt 22%Cookie consent banner 19%Ad pixels 23%llms.txt 15%HTTPS homepage 98%DMARC record 64%
TNP.media
Get in touch
TNP.media
Get in touch
  1. Home
  2. Services
  3. AI crawler policy
3–5 daysPolicy, configs

AI crawler policy

Decide which AI bots may use your content — and enforce it.

What's included

  • Bot traffic review
  • robots.txt rules
  • llms.txt
  • Server-side blocking options
  • Monitoring

Steps

  1. 1Review
  2. 2Policy
  3. 3Implement
  4. 4Monitor

What we see in Global

10%
Blocks an AI crawler
9,012 / 85,917
9.6%
Blocks GPTBot
8,222 / 85,917
15%
llms.txt
10,279 / 68,769
AI crawlers
Timeline3–5 days
FormatsPolicy, configs
PriceFixed quote after the brief
For
PublishersMediaE-commerce
Request a quote

Free tools for this

robots.txt generatorCrawler rules incl. AI bots
AI crawler policy

Tell us about your task

A couple of sentences is enough. We reply within one business day.

We only use your email to reply.

Other services

Digital PRData-driven stories that earn coverage and links.Story research from dataPress materialsMedia outreach DetailsMedia monitoringMentions of your brand and topics across news sites and feeds.Source listKeyword trackingDaily digest DetailsAd operationsPixels, conversions and campaign tagging that actually match up.Pixel and conversion auditServer-side eventsUTM conventions DetailsPublisher ad tech auditads.txt, consent, tags and page weight — what costs you revenue.ads.txt and sellers.json checkConsent and TCF setupTag and bidder audit Details