- Home
- Methodology
How we measure
Every number on this site comes from our own crawler. Here is what it does and where its limits are.
Samples
| Market | Rule | Sites | Crawled | Homepage parsed |
|---|---|---|---|---|
| Global | Top 25,000 of the Tranco list | 100,000 | 100,000 | 68,769 |
| Netherlands | Top .nl domains | 17,114 | 17,114 | 8,580 |
| Japan | Top .jp domains | 16,717 | 16,717 | 12,114 |
| Italy | Top .it domains | 15,096 | 15,096 | 7,546 |
| United Kingdom | Top .uk domains | 27,305 | 27,305 | 11,909 |
| Spain | .es domains | 5,582 | 5,582 | 4,497 |
| Latin America | .mx .ar .cl .pe .uy .com.co … and top .br | 27,231 | 27,231 | 20,254 |
What we check on each site
- DNS: A/AAAA, NS, MX, TXT (SPF and SaaS verification), DMARC, CAA, DNSSEC, BIMI, MTA-STS.
- TLS: certificate authority, validity, protocol version — with a direct handshake.
- Homepage: technologies (500 signatures), SEO tags, Schema.org, page weight, third parties.
- Response headers: security headers, compression, HTTP version, cookies.
- robots.txt (incl. AI crawlers), sitemap.xml, llms.txt, security.txt, ads.txt.
- One visit per site and file; robots.txt is respected; bot challenges are recorded, not bypassed.
Sources
Tranco746PX
DateOct 21, 2026
IANA1,437 zones
Lists: Tranco (research ranking of popular domains), IANA root zone, Public Suffix List, iptoasn.com.