# How to Block AI Crawlers with robots.txt Without Leaving AI Search > Add a robots.txt group for each AI crawler you want to stop, such as GPTBot, ClaudeBot or CCBot, with a Disallow: / rule. Disallow Google-Extended and Applebot-Extended to opt out of AI training without leaving search. robots.txt is only a request: some user-triggered fetchers may ignore it, and a named group stops following your general rules. - Canonical URL: https://backlnk.com/blog/block-ai-crawlers-robots-txt - Byline: María Graciela - First published: 2026-11-01 - Topics: AI Visibility, Technical SEO - Evidence: external research (cited); editorial interpretation; hypothetical, illustrative examples (labelled) — not real results AI companies run separate bots for training, AI search and user requests. Here are the documented tokens, how robots.txt matching works and copy-ready configurations. "Block AI crawlers" sounds like a single switch. It is not. Most AI companies now run separate bots for model training, for their AI search features and for fetching a page when a user asks about it, and each bot has its own robots.txt token. Block the wrong one and you can disappear from an AI search product while another token still lets your pages be collected for training. robots.txt remains the control nearly every vendor documents, but it has sharp edges. A crawler named in its own group stops following your general rules, several vendors say their user-triggered fetchers may ignore the file, and Google's AI training control is a token that never crawls at all. This guide lists the tokens each vendor documents, as reviewed on October 31, 2026, explains how a crawler picks its rules under RFC 9309, and gives copy-ready configurations for the most common goals, together with the limits to understand before you deploy them. **Key takeaways** - Decide per purpose. OpenAI, Anthropic, Perplexity, Meta, Amazon and Mistral document separate tokens for different jobs, such as training, AI search and user-triggered fetching. - A crawler named in its own group ignores your User-agent: * rules, so repeat any general rules it should still follow. - Google-Extended and Applebot-Extended are control tokens: they opt you out of AI training uses without removing you from Google Search or Apple's search features. - Google's AI Overviews and AI Mode are part of Search. They follow Googlebot rules and snippet controls, not Google-Extended. - robots.txt is a request, not access control, and some user-triggered fetchers are documented as not following it. - Non-standard noai meta tags are not among Google's documented robots rules, and Meta says it relies on robots.txt instead. ## Training, search, user requests: the four kinds of AI tokens Before you write a single rule, sort the bots by what they do. The vendors' own documentation separates four roles, and the right configuration depends on which of them you want to stop. - **Training crawlers** — Collect pages that may be used to train models, for example GPTBot, ClaudeBot, meta-externalagent and MistralAI-Training. CCBot builds Common Crawl's open repository of web data, which anyone can access and analyze. - **AI search crawlers** — Index pages so an AI search or answer product can surface and link them, for example OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot. - **User-triggered fetchers** — Visit a page because a person asked an assistant about it, for example ChatGPT-User, Claude-User and Perplexity-User. Several vendors document that these may not follow robots.txt. - **Control tokens** — `Google-Extended` and `Applebot-Extended` never crawl. They are read from robots.txt to decide how content fetched by the vendor's main crawler may be used. ## AI and search crawler tokens by vendor The table summarizes each vendor's official crawler page. The robots.txt token is the name you put after `User-agent:`. | Company | Token | Purpose | Documented robots.txt behavior | | --- | --- | --- | --- | | OpenAI | `GPTBot` | Training | Disallowing it indicates content should not be used to train OpenAI's foundation models | | OpenAI | `OAI-SearchBot` | ChatGPT search | Opted-out sites are not shown in ChatGPT search answers but can still appear as navigational links; about 24 hours to take effect | | OpenAI | `ChatGPT-User` | User-triggered | Robots.txt rules may not apply; not used to decide search inclusion | | Anthropic | `ClaudeBot` | Training | A block signals that the site's future materials should be excluded from training datasets | | Anthropic | `Claude-SearchBot` | Search | Disabling it prevents indexing for search and may reduce visibility in Claude's search results | | Anthropic | `Claude-User` | User-triggered | Disabling it prevents retrieval for user queries; Anthropic says its bots honor robots.txt and support Crawl-delay | | Google | `Googlebot` | Search, including AI Overviews and AI Mode | Always obeys robots.txt when crawling automatically; its rules are the control for AI features in Search | | Google | `Google-Extended` | Control token | Never crawls; governs use for future Gemini training and for grounding in Gemini Apps and Vertex AI; no effect on Search inclusion or ranking | | Apple | `Applebot` | Spotlight, Siri and Safari search; data may also train Apple models | Follows rules aimed at Applebot, and Googlebot rules when Applebot is not named; ignores crawl-delay | | Apple | `Applebot-Extended` | Control token | Does not crawl; disallowing it opts out of training Apple's foundation models while staying in search | | Perplexity | `PerplexityBot` | Search | Not used to crawl for foundation models; Perplexity recommends allowing it to appear in results | | Perplexity | `Perplexity-User` | User-triggered | Generally ignores robots.txt because a user requested the fetch | | Meta | `meta-externalagent` | Training and product indexing | Blocked with a Disallow rule; robots.txt may be cached for up to 24 hours | | Meta | `meta-webindexer` | Meta AI search | Allowing it helps Meta cite and link content in Meta AI responses | | Meta | `meta-externalfetcher` | User-triggered | May bypass robots.txt | | Meta | `facebookexternalhit` | Link previews | May bypass robots.txt for security or integrity checks | | Amazon | `Amazonbot` | Amazon products; may train Amazon AI models | Honors user-agent and allow/disallow rules; may use a robots.txt copy cached for up to 30 days | | Amazon | `Amzn-SearchBot` | Search experiences such as Alexa | Not used for generative AI training; follows other search bots' rules if not named | | Amazon | `Amzn-User` | User-triggered | May not follow all robots.txt directives | | Common Crawl | `CCBot` | Open web dataset | Blocked with `Disallow: /`; Common Crawl warns that other crawlers impersonate CCBot | | Mistral AI | `MistralAI-Training` | Training | Can be disallowed; not used for search or live answers | | Mistral AI | `MistralAI-Index` | Search | Not used for generative AI training | | Mistral AI | `MistralAI-User` | User-triggered | Governs which sites user requests can reach | | DuckDuckGo | `DuckAssistBot` | AI-assisted answers | Not used for training; an opt-out takes effect after 72 hours and does not affect organic rankings | | Microsoft Bing | `bingbot` | Search | Bing's September 2023 announcement controls use in Bing Chat and model training with NOARCHIVE and NOCACHE page tags, not a separate token | > **Note:** **Tokens change.** Vendors add, split and rename bots. Re-check the official pages linked in the sources before relying on this table, and note the review date in your own documentation. Google's user-triggered fetchers, such as Google-Agent, are listed separately by Google, which says they generally ignore robots.txt. ## How robots.txt decides which rules a crawler follows [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), the IETF standard for robots.txt, defines a group as one or more `User-agent` lines followed by rules. A crawler looks for the group that names its product token, matching case-insensitively, and merges all groups that name it. Only when no group names it does it fall back to the `User-agent: *` group; with no such group, no rules apply. Inside the chosen group the most specific, longest matching path wins, and when an Allow and a Disallow rule are equivalent, Allow should be used. [Google's interpretation](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) follows the same precedence and states that user-agent-specific groups and the `*` group are never combined. ### The trap: a named group replaces your general rules *Illustrative example:* A site at example.com keeps checkout and internal search out of every crawler with a `*` group, then adds a GPTBot group to protect a premium folder. GPTBot now follows only its own group, so it may crawl the checkout and search pages the site meant to exclude. *robots.txt (problem: GPTBot ignores the * group)* ``` User-agent: * Disallow: /checkout/ Disallow: /search/ User-agent: GPTBot Disallow: /premium/ ``` *robots.txt (fixed: general rules repeated in the named group)* ``` User-agent: * Disallow: /checkout/ Disallow: /search/ User-agent: GPTBot Disallow: /premium/ Disallow: /checkout/ Disallow: /search/ ``` ### Status codes, caching and scope The file's HTTP status matters as much as its content. Under RFC 9309, a 4xx response means crawlers may access everything, while a 5xx or network error means they must assume complete disallow. Google adds detail: it treats 4xx codes other than 429 as if no file existed, and on server errors it stops crawling for the first 12 hours, then relies on the last good copy for up to 30 days. A robots.txt that returns server errors can therefore halt compliant crawlers, Googlebot included. Changes are not instant. The RFC says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. OpenAI and Perplexity mention about 24 hours, Meta up to 24 hours, DuckDuckGo 72 hours, and Amazon says its crawlers may use a copy from the last 30 days. Rules apply only to the host, protocol and port that serve the file, so every subdomain needs its own robots.txt; Anthropic explicitly asks you to repeat opt-outs for each subdomain. Crawlers must parse at least the first 500 KiB of the file. ## Copy-ready robots.txt configurations Add these groups to your existing file and keep your own `User-agent: *` group, shown here with an example rule. Group order does not matter. Crawlers you do not name keep following the `*` group. ### 1. Opt out of AI training, stay in search and AI search *robots.txt: block training only* ``` # AI training crawlers User-agent: GPTBot User-agent: ClaudeBot User-agent: meta-externalagent User-agent: MistralAI-Training User-agent: CCBot User-agent: Amazonbot Disallow: / # Control tokens: no crawling, only usage decisions User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / # Everyone else, including search and AI search crawlers User-agent: * Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml ``` Two entries deserve a decision rather than a reflex. CCBot builds an open dataset used for many purposes, not only training, so include it if you do not want your pages in that dataset. Amazonbot also improves Amazon products and services, so blocking it may affect more than training; Amazon does not specify which products. ### 2. Block AI training and AI search, keep classic search *robots.txt: block training, AI search and AI fetchers* ``` # AI training crawlers and control tokens User-agent: GPTBot User-agent: ClaudeBot User-agent: meta-externalagent User-agent: MistralAI-Training User-agent: CCBot User-agent: Amazonbot User-agent: Google-Extended User-agent: Applebot-Extended # AI search crawlers User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: meta-webindexer User-agent: Amzn-SearchBot User-agent: MistralAI-Index User-agent: DuckAssistBot # User-triggered fetchers (several may not follow robots.txt) User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: meta-externalfetcher User-agent: Amzn-User User-agent: MistralAI-User Disallow: / User-agent: * Disallow: /checkout/ ``` This configuration leaves Googlebot, Applebot and bingbot alone, because they power classic search. It cannot remove you from Google's AI Overviews or AI Mode without also removing you from Google Search: Google documents Googlebot rules, `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex` as the controls for its AI features. Apple likewise documents `nosnippet` for keeping content out of answers its AI models generate in Siri and Search. ### 3. Keep one section out of AI training *robots.txt: protect one folder* ``` User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: /research/ Allow: /research/summaries/ Disallow: /checkout/ User-agent: * Disallow: /checkout/ ``` The `Allow` line wins for /research/summaries/ because it is the longer match, and `/checkout/` is repeated because the named group no longer inherits it from the `*` group. ## Control tokens: Google-Extended and Applebot-Extended [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) has no user agent of its own. Crawling is done by Google's existing crawlers, and the token only tells Google whether that content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states that it does not affect a site's inclusion in Google Search and is not a ranking signal. > **Common myth:** **"Disallowing Google-Extended keeps my site out of AI Overviews."** It does not. Google's [AI features page](https://developers.google.com/search/docs/appearance/ai-features) explains that AI is part of Search, so Googlebot rules control crawling for AI Overviews and AI Mode, and snippet controls or `noindex` limit what is shown. On October 30, 2026, Google announced plans to support a shared AI training control from the IETF AIPREF working group, a policy token in robots.txt that Google's crawler documentation writes as `user-agent: __usage_ai-training`. If a site opts out with it and does not name Google-Extended, Google plans to treat that as disallowing Google-Extended. Google aims to support it in 2027, so do not rely on it yet; Google says existing controls such as Google-Extended will keep working. [Applebot-Extended](https://support.apple.com/en-us/119829) works the same way for Apple. It does not crawl, and disallowing it opts your content out of training Apple's foundation models, while pages remain eligible for Spotlight, Siri and Safari search. Apple adds two page-level signals: `nosnippet` keeps content from being used as context for AI-generated answers, and `isAccessibleForFree: false` in structured data does the same for paywalled pages. ## What robots.txt cannot do - Enforce anything. RFC 9309 states that its rules are not a form of access authorization, and Google notes that compliance is up to each crawler. Use authentication for anything private; listing a path in robots.txt also makes it public. - Stop user-triggered fetchers reliably. OpenAI, Perplexity, Meta, Amazon and Google document that some fetchers acting for a user may not follow robots.txt. - Reach back in time. Anthropic describes a ClaudeBot block as covering the site's future materials, and none of the vendor pages we reviewed present robots.txt as a way to remove data already collected. - Remove a URL from Google Search. A disallowed URL can still be indexed without its content if other pages link to it. Use noindex and keep the page crawlable; our guide to pages that are not indexed explains the difference. - Prove who is crawling. User-agent strings can be spoofed. OpenAI, Perplexity, Anthropic, Apple, Amazon and Common Crawl publish IP ranges or reverse DNS so you can verify requests. If you add firewall rules, note Anthropic's warning that blocking its IP addresses can stop it from reading your robots.txt opt-out. ### Page-level signals and non-standard noai tags Some vendors document page-level controls. Google's `nosnippet` also prevents content from being used as direct input for AI Overviews and AI Mode; Apple honors `nosnippet` for its AI answers; Amazon treats `noarchive` as "do not use the page for model training"; and Bing's 2023 announcement uses NOARCHIVE and NOCACHE. Google notes that such settings are read only when crawlers may access the page, and the same logic applies to any crawler: a robots.txt block hides the meta tag from the bot it targets. Values such as `noai` and `noimageai` are different. They are not among the rules in [Google's robots meta tag specification](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), Meta writes that it relies on robots.txt "rather than non-standard formats like NoAI tags", and none of the other vendor pages we reviewed document honoring them. Also be careful with `facebookexternalhit`: blocking it can break link previews, covered in our [Open Graph guide](https://backlnk.com/blog/open-graph-tags). ## Step-by-step: block AI crawlers safely 1. **Decide per purpose.** Write down whether you want to allow training, AI search and user-triggered fetches, vendor by vendor. Mixed answers are normal. 2. **Collect every robots.txt you serve.** Check the main host and each subdomain, because each file covers only its own host. 3. **Add one group per decision.** Name the tokens from the table, and copy into each named group any general rules it should keep. 4. **Add control tokens.** Use `Google-Extended` and `Applebot-Extended` for training opt-outs that keep you in search. 5. **Deploy and confirm a 200 response.** Under RFC 9309 a file that returns 5xx means complete disallow, and Google pauses crawling while it retries. 6. **Allow time.** Expect roughly a day for most vendors, 72 hours for DuckDuckGo and longer for crawlers that cache the file. 7. **Verify.** Check which group and line each crawler now matches, then confirm in your logs, using published IP ranges to tell real bots from impostors. ## Check your setup with the AI Crawler Access Checker The [AI Crawler Access Checker](https://backlnk.com/tools/ai-crawler-access-checker) reads your robots.txt the way RFC 9309 defines it and evaluates every crawler in its reviewed list, from GPTBot, ClaudeBot and PerplexityBot to Google-Extended, CCBot and bingbot, for the site root and for the URL you submit. For each one it shows allowed or blocked, grouped by company and purpose, with the exact group and line that decided, and it applies documented fallbacks such as Applebot following Googlebot rules. The report also flags named groups that no longer inherit your `*` rules, combinations that look unintended (such as a blocked search bot next to an allowed training bot from the same vendor), unreachable or oversized robots.txt files, and the page's own meta robots and X-Robots-Tag directives, including non-standard `noai` values. It shows what you ask crawlers to do, not what they actually do, and it says so: vendor names and purposes follow each vendor's documentation as reviewed on the date shown in the report. Crawler access is also part of the free [full website audit](https://backlnk.com/tools/full-audit), delivered by email. ## Next step Decide which of the three purposes you want to allow, copy the matching configuration above, and run your site through the [AI Crawler Access Checker](https://backlnk.com/tools/ai-crawler-access-checker) after deployment to confirm that every token lands on the rule you intended. ## Frequently asked questions ### Does blocking GPTBot remove my site from ChatGPT search? No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, and each setting is independent. To leave ChatGPT search answers you must disallow OAI-SearchBot, and OpenAI says opted-out sites can still appear as navigational links. ### Does disallowing Google-Extended remove my content from AI Overviews? No. Google-Extended controls use for Gemini training and for grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode are part of Google Search, which follows Googlebot rules and snippet controls such as nosnippet. ### Do AI crawlers have to obey robots.txt? No. RFC 9309 states that its rules are not a form of access authorization, so compliance is up to each crawler. Most vendors document that their automated crawlers follow robots.txt, but several say their user-triggered fetchers may not. Use authentication for real access control. ### How long does a robots.txt change take to apply? It depends on the vendor. RFC 9309 says a cached copy should not be used for more than 24 hours unless the file is unreachable. OpenAI and Perplexity mention about 24 hours, DuckDuckGo 72 hours, and Amazon may use a copy up to 30 days old. ### Do noai meta tags block AI crawlers? None of the vendors whose documentation we reviewed say they honor noai or noimageai. These values are not among Google's documented robots meta rules, and Meta says it relies on robots.txt rather than non-standard formats like NoAI tags. ## Sources - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html), IETF RFC Editor (accessed 2026-10-31) - [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec), Google for Developers (accessed 2026-10-31) - [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro), Google Search Central (accessed 2026-10-31) - [List of Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), Google for Developers (accessed 2026-10-31) - [List of Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers), Google for Developers (accessed 2026-10-31) - [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), Google Search Central (accessed 2026-10-31) - [Robots meta tag, data-nosnippet, and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag), Google Search Central (accessed 2026-10-31) - [Supporting a shared AI training control for websites](https://developers.google.com/search/blog/2026/10/ai-training-control), Google Search Central Blog (accessed 2026-10-31) - [Overview of OpenAI Crawlers](https://platform.openai.com/docs/bots), OpenAI (accessed 2026-10-31) - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), Anthropic (accessed 2026-10-31) - [About Applebot](https://support.apple.com/en-us/119829), Apple (accessed 2026-10-31) - [Perplexity Crawlers](https://docs.perplexity.ai/guides/bots), Perplexity (accessed 2026-10-31) - [Meta Web Crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers), Meta for Developers (accessed 2026-10-31) - [Amazon Web Search crawlers](https://developer.amazon.com/amazonbot), Amazon (accessed 2026-10-31) - [CCBot](https://commoncrawl.org/ccbot), Common Crawl (accessed 2026-10-31) - [Mistral crawlers](https://docs.mistral.ai/robots), Mistral AI (accessed 2026-10-31) - [Is DuckAssistBot related to DuckDuckGo?](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot), DuckDuckGo (accessed 2026-10-31) - [Announcing new options for webmasters to control usage of their content in Bing Chat](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat), Microsoft Bing Blog (accessed 2026-10-31)