Skip to content
Backlnk
Free audit

Open data · CC BY 4.0

AI crawler access in the Tranco top 10,000

What the robots.txt files of the 10,000 most popular domains ask of GPTBot, ClaudeBot, Google-Extended, CCBot and 21 other crawler tokens — one row per domain, measured on 31 October 2026. Free to reuse, including commercially, with credit.

domains (Tranco top 10,000)
10,000domains (Tranco top 10,000)
crawler tokens from 11 companies
25crawler tokens from 11 companies
domains in the analysis base
5,996domains in the analysis base
columns, one row per domain
38columns, one row per domain
Read the study: findings, method and charts

Get the dataset

CSV, README and licence in one ZIP (252 KB). We e-mail you a personal download link.

The link works for seven days.

How we handle your data (opens in a new tab)

  • Free
  • CC BY 4.0
  • Commercial use allowed

What's inside

One row per domain, a decision for every crawler

A tidy CSV ready for a spreadsheet, SQL or pandas, with a README that explains every column, the method and the limitations.

  • The domain

    rank · domain

    Its position in Tranco list 746VX and the domain as listed.

  • What the server answered

    robots_state · http_status · final_url · bytes · served_html

    Found, missing, unreachable or not readable, the last HTTP status, the robots.txt URL after redirects and the size that was parsed.

  • How the file is written

    groups · rules · wildcard_blocks_root

    Number of groups and rules, and whether the * group blocks the whole site.

  • A decision for every crawler

    GPTBot … MistralAI-User (25 columns)

    allowed, blocked or unknown for the site root /, evaluated under RFC 9309 — from OpenAI, Anthropic, Google, Microsoft, Apple, Perplexity, Meta, Amazon, Common Crawl, DuckDuckGo and Mistral AI.

  • Summaries

    blocked_tokens · named_tokens · measured_at

    Which tokens are blocked, which are named in their own group, and when the request was made.

rankdomainrobots_statehttp_statusGPTBotOAI-SearchBotClaudeBotGoogle-ExtendedCCBotGooglebot
1google.comfound200allowedallowedallowedallowedallowedallowed
5facebook.comfound200blockedblockedblockedblockedblockedallowed
9microsoft.comfound200allowedallowedallowedallowedallowedallowed
14instagram.comfound200blockedblockedblockedblockedblockedallowed
Four real rows and ten of the 38 columns, as measured on 31 October 2026. The download has all 10,000 rows.

Licence

Free to reuse under CC BY 4.0

The Creative Commons Attribution 4.0 International licence is the standard for open research data. The full legal code is in the download.

You may

  • Copy and share the data
  • Adapt it, combine it and build on it
  • Use it for any purpose, including commercially

As long as you

  • Credit Backlnk and the dataset
  • Link to the licence
  • Say if you changed the data

Third-party data

  • Which domains are included and their rank come from the Tranco list 746VX
  • Tranco ranks have their own terms; cite Tranco if you republish them
  • The README explains the details
Suggested credit

Data: Backlnk, “AI crawler access in the Tranco top 10,000” (2026), https://backlnk.com/research/ai-crawler-access-2026-11, licensed under CC BY 4.0.

Citation

Backlnk (2026). AI crawler access in the Tranco top 10,000: robots.txt survey of Tranco list 746VX, measured 2026-10-31 (version 1.0) [Data set]. https://backlnk.com/research/ai-crawler-access-2026-11. CC BY 4.0.

Method in brief

Measured once, read by the rules

  1. 1

    A fixed, citable sample

    The top 10,000 domains of Tranco list 746VX (created 30 October 2026), a research ranking that combines five popularity lists.

  2. 2

    One request per domain

    robots.txt fetched once from each domain's HTTPS origin on 31 October 2026, 15:05–15:13 UTC, with a clearly named user agent.

  3. 3

    Read the way crawlers must

    Status codes, groups and rules evaluated under RFC 9309 for 25 tokens, each checked against its operator's documentation.

  4. 4

    A transparent analysis base

    5,996 domains with a readable file or a clean 404/410; everything else stays in the data, labelled, so you can choose differently.

A robots.txt file states what a site asks; it says nothing about whether a crawler complies. One snapshot, one vantage point, root path only — the README lists every limitation.

Full method

Frequently asked questions

Is the dataset free?
Yes. Downloading costs nothing, and you may reuse the data for any purpose, including commercially, under the CC BY 4.0 licence.
Why do I need to enter my e-mail address?
We send you a personal download link instead of publishing the file. The link works for seven days; if you didn't ask for it, the e-mail can simply be ignored and nothing else arrives.
What do you do with my e-mail address?
We use it to send your download link. See our privacy page for details.
How do I credit Backlnk?
Name Backlnk and the dataset, link to this page, link to the licence and say if you changed the data. The suggested credit line above does all of that; it also works as a caption under a chart.
Can I share the file or publish my own analysis?
Yes. CC BY 4.0 allows sharing and adapting the data as long as you give credit. The Tranco ranks in the file are third-party data with their own terms; the README explains what that means if you republish them.
How was the data collected?
One request for robots.txt per domain on 31 October 2026, evaluated under RFC 9309 for the site root. The study explains the method, definitions and limitations in full, and the README in the download summarises them with a dictionary of every column.

What does your robots.txt tell AI crawlers?

Run the same evaluation on your own site with the free AI Crawler Access Checker: every token, the rule that decides it, and what to change.

Check my site