Who Blocks AI Crawlers? We Checked robots.txt on the Top 10,000 Sites

Site owners now face a decision that barely existed a few years ago: which AI companies may read their pages, and for what. Many AI vendors document separate crawlers for model training, for AI search and for fetching a page when a user asks about it, each with its own robots.txt token. So what have the most visited sites actually decided?

To find out, Backlnk requested the robots.txt file of every domain in the Tranco top 10,000 on October 31, 2026, and evaluated each answer the way RFC 9309 defines it for 25 crawler tokens from 11 companies. In all, 5,996 domains gave a readable answer. Every percentage below is a share of those reachable top-10,000 sites, and the full dataset is available to download.

What we measured and why

robots.txt is where a site states which crawlers may fetch which paths. It is not access control: RFC 9309, the IETF standard, says its rules "are not a form of access authorization." But 10 of the 11 companies in our crawler catalog document it as the control for their crawlers, which makes it the most direct public record of what sites ask AI companies to do. The study asks: which sites in the Tranco top 10,000 block AI crawlers from their entire site, and which crawlers do they block?

• Sample. The top 10,000 domains of Tranco list 746VX, created on October 30, 2026 from CrUX, Farsight, Majestic, Cloudflare Radar and Cisco Umbrella data for October 1 to 30.
• Measurement. One request per domain on October 31, 2026, between 15:05 and 15:13 UTC, from a single network vantage point.
• Tokens. 25 tokens from 11 companies, reviewed against each company's documentation that day: 8 for AI training and datasets (two of them control tokens), 7 AI search crawlers, 7 user-triggered fetchers and 3 search engines.
• Decision. A token is blocked when, under RFC 9309 group matching and longest-match rules, it may not fetch the site root /. Rules that close only some directories do not count.

The analysis base is 5,996 domains: 5,086 served a robots.txt file, 451 served an HTML page in its place (no rules) and 459 answered 404 or 410 (no file). The rest, mainly 3,205 domains that did not answer and 701 that refused the request, are excluded.

One in five top sites blocks AI training crawlers

The most-blocked tokens all belong to training and dataset crawlers: Common Crawl's CCBot at 16.9% (1,015 sites), OpenAI's GPTBot at 15.8% (945) and Anthropic's ClaudeBot at 15.1% (906). In total, 20.0% of sites block at least one of the eight training and dataset tokens, but only 6.0% (361) block all eight. AI search crawlers and user-triggered fetchers sit lower, topped by PerplexityBot (9.7%) and ChatGPT-User (8.3%), and the classic search engines are barely touched: Googlebot 1.8%, bingbot 2.2%, Applebot 3.4%.

Training, AI search and user-triggered fetchers

AI training and datasets

Seven of the eight tokens sit within four points of each other, from 13.2% for Amazonbot to 16.9% for CCBot. Two are broader than training: Common Crawl builds an open repository of web crawl data that anyone can access and analyze, and Amazon says Amazonbot improves its products and services and may be used to train Amazon AI models. The outlier is MistralAI-Training at 6.5%: only 3.9% of robots.txt files name it, and almost half of its blocks (194 of 390) come from blanket User-agent: * rules.

AI search crawlers

These crawlers decide whether a site can be found and linked in an AI product's answers. At least one of the seven is blocked by 11.3% of sites (678; 95% CI 10.5–12.1%). For the less familiar names, many blocks are incidental: two thirds of Amzn-SearchBot's blocks (188 of 281) and four in five of MistralAI-Index's (194 of 240) come from blanket rules rather than from a rule that names the crawler.

User-triggered fetchers

These fetchers visit a page because a person asked an assistant about it. At least one of the seven is blocked by 10.5% of sites (632; 95% CI 9.8–11.3%). Read these figures with the vendors' caveats: OpenAI says robots.txt rules "may not apply" to ChatGPT-User, Perplexity says Perplexity-User "generally ignores robots.txt rules", and Meta and Amazon give similar warnings for their user-triggered fetchers. For those tokens, a block is a stated preference. Anthropic and Mistral AI, by contrast, present Claude-User and MistralAI-User as the controls for which sites user requests may reach.

Vendor by vendor: training no, search yes

OpenAI, Anthropic, Meta, Amazon and Mistral AI run separate training and AI search crawlers; Google and Apple pair a training control token with their search crawler. Compared site by site, the pairs show what most blockers want: out of training, still in search.

OpenAI documents its two settings as independent, and sites use them that way: 556 of the 945 sites that block GPTBot (58.8%; 95% CI 55.7–61.9%) still allow OAI-SearchBot, while only one site blocks OAI-SearchBot without GPTBot. Anthropic looks almost identical, with 519 of 906 ClaudeBot blockers (57.3%) leaving Claude-SearchBot allowed. The split is widest where the search side is a classic search engine: 87.2% of Google-Extended blockers leave Googlebot alone, and 76.7% of Applebot-Extended blockers leave Applebot alone.

Mistral AI is the exception. More sites block both of its tokens (207) than only MistralAI-Training (183), largely because MistralAI-Index is named in just 49 robots.txt files, so most of its blocks come from the * group.

Across all tokens the picture is more mixed: 547 of the 1,197 sites that block a training or dataset token (45.7%) block no AI search crawler at all, and the rest block at least one. Search-only blocking, however, is almost absent: only 28 of the 678 sites that block an AI search crawler leave every training token allowed.

Top-ranked sites block more AI search crawlers

Splitting the sample by rank shows a gap for AI crawlers and no clear difference for search engines. In the Top 1,000 (analysis base 588), 24.3% of sites block at least one AI training or dataset token, against 19.5% of sites ranked 1,001–10,000 (base 5,408), and the two intervals do not overlap. The gap is wider for AI search crawlers, 16.3% against 10.8%, and user-triggered fetchers, 15.6% against 10.0%. For search engines the shares are close, 3.2% against 3.8%, and the intervals overlap.

Part of the difference comes from blanket blocks, twice as common at the top (6.0% of Top 1,000 sites against 3.0%). Counting only rules that name a token, the AI search gap remains clear at 12.2% against 8.1%, while the training gap narrows to 20.7% against 16.9%, with overlapping intervals. The Top 1,000 base is also small: 412 of those domains are excluded, most because they did not answer.

Control tokens: out of AI training, still in search

Google-Extended and Applebot-Extended never crawl. Google describes Google-Extended as a standalone product token that manages whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI, and says it does not affect inclusion in Google Search. Apple says Applebot-Extended only determines how data crawled by Applebot is used, and pages that disallow it can still appear in search results. Sites use these controls far more often than they block the crawlers behind them:

Applebot is the one search crawler that sites block by name with some regularity: 92 sites, against 9 for Googlebot and 16 for bingbot. Another 8 block it only through Apple's documented fallback: they block Googlebot by name and never mention Applebot, which then follows the Googlebot rules.

How sites block: by name or with User-agent: *

Most AI blocks are explicit: they come from a group that names the token. For GPTBot, 797 of the 945 blocks (84.3%) name it; the other 148 come from the User-agent: * group, which applies to every crawler without a group of its own. Overall, 17.8% of sites (1,070) block at least one of the 22 AI tokens by name.

A smaller group takes the blunt route. 196 sites (3.3%; 95% CI 2.8–3.7%) disallow the root in their * group, which blocks every crawler they do not name, search engines included. That is where most search engine blocks come from: 101 of the 110 Googlebot blocks and 117 of the 133 bingbot blocks. Of those 196 sites, 90 add a group that lets Googlebot back in; the other 106 block it too.

Two templates stand out. The most common combination, on 162 sites (2.7%), blocks exactly the eight training and dataset tokens and none of the other 14 AI tokens. The second, on 129 sites (2.2%), blocks all 22 AI tokens, and 128 of those also disallow the root in their * group.

The other tokens sites name

Sites also name crawlers outside our 25 tokens. We count them but do not evaluate them, because the catalog includes only tokens we could verify on the operator's own documentation. Among the 5,086 robots.txt files in the base, 95.6% have a * group, and these uncatalogued tokens appear most often:

anthropic-ai and claude-web are not among the three bots (ClaudeBot, Claude-User and Claude-SearchBot) that Anthropic lists on the help page we reviewed on October 31, 2026, and Bytespider is not in the catalog because we could not verify it on an official vendor page. Naming is also not the same as blocking: of the 1,128 files with a GPTBot group, 335 still let it fetch the root, either by allowing it or by restricting only some paths.

What this means for your site

• Decide per purpose, not per company. For six of seven companies, most top sites that block the training token leave the search crawler allowed. Our guide on how to block AI crawlers with robots.txt has copy-ready configurations for that and other goals.
• Use control tokens for Google and Apple training opt-outs. Both companies say disallowing Google-Extended or Applebot-Extended does not remove a site from their search results.
• Treat blocks of user-triggered fetchers as requests. Several vendors say these fetchers may not follow robots.txt, and RFC 9309 points to valid security measures, such as HTTP authentication, for access control.
• Know what llms.txt does not do. It contains no access rules, as our explainer on what llms.txt is shows. If your goal is to be quoted, see how to get cited in AI answers.
• Check your own file. The AI Crawler Access Checker applies the same RFC 9309 evaluation used in this study and shows the group and line behind each verdict. It is also part of the free full website audit.

Methodology and limitations

Each domain received one HTTPS request for /robots.txt from BacklnkBot/1.0 through Backlnk's egress proxy, which recorded zero refusals during the run. The full methodology follows the article.

Exclusions. 3,205 domains were unreachable; Tranco ranks domains, not websites, and the highest-ranked unreachable entries are largely CDN, DNS and cloud infrastructure domains. Another 701 refused the request (642 with HTTP 403), and 98 were non-public, unreadable or invalid. RFC 9309 lets crawlers treat a 4xx answer as "allow everything"; counting the 701 refusals that way (base 6,697) lowers each rate slightly, for example GPTBot from 15.8% to 14.1%.

• One snapshot. One day, one vantage point, one user agent. Sites can serve different files to different clients, and files change.
• Root only. A token barred from some directories but allowed at / counts as allowed.
• Stated rules, not behavior. We did not observe crawler compliance, and robots.txt cannot show firewall blocks, robots meta tags or X-Robots-Tag headers.
• A defined population. The results describe the Tranco top 10,000, not the web. The 95% Wilson intervals express statistical uncertainty, not representativeness.
• Catalog choices. We applied Apple's documented Googlebot fallback for Applebot, but not Amazon's fallback for Amzn-SearchBot, because Amazon does not say which search bots' rules it follows. Bing's robots.txt documentation could not be re-verified on review day.

Download the data

The dataset has one row for each of the 10,000 domains: rank, fetch outcome, HTTP status, final URL, file size, group and rule counts, an allowed, blocked or unknown decision for each of the 25 tokens, the lists of blocked and named tokens, and the measurement time. Filter on robots_state and http_status to reproduce the analysis base. Unreachable domains show every token as blocked, the RFC 9309 reading of an unreachable file, and are excluded from every percentage here.

It comes as one ZIP file with the CSV, a README (method summary and a dictionary of every column) and the licence. Get it free on the dataset page: enter your e-mail address and we send you a personal download link. The data is licensed under Creative Commons Attribution 4.0 (CC BY 4.0), so you can reuse it for any purpose, including commercially, as long as you credit Backlnk. The ranks come from the Tranco list and keep their own terms.