Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot and OAI-SearchBot are separate tokens, and OpenAI documents each one. Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Does blocking Google-Extended affect Google Search?

No. Google-Extended is a control token, not a crawler. Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal. Crawling itself is done with existing Google user agents.

Why does the builder sometimes add an Allow group for Applebot?

Applebot respects robots.txt directives targeted at Applebot. If robots instructions don't mention Applebot but mention Googlebot, Applebot follows the Googlebot instructions. When you block Googlebot but keep Applebot, the builder adds a group that names Applebot with "Allow: /" so it does not inherit the block. Under RFC 9309 a crawler named by a group ignores the "*" group, so copy any "*" rules you still want it to follow into that group.

Do AI assistants follow robots.txt when a user asks them to open a page?

Not always. The vendors of ChatGPT-User, Perplexity-User, meta-externalfetcher, facebookexternalhit and Amzn-User document that these user-triggered fetchers may not follow robots.txt. Claude-User and MistralAI-User are documented without that exception.

My robots.txt already has a group for one of these crawlers. Can I just add the new lines?

Edit the existing group instead. Under RFC 9309 a crawler merges every group that names it, the longest matching rule wins, and an Allow rule wins a tie, so an existing "Allow: /" for the same token would override the new "Disallow: /".

Is robots.txt enough to stop AI from using my content?

It is a request, not a lock. RFC 9309 states that its rules are not a form of access authorization: crawlers that honour the protocol follow them, others may not. Content that must stay private needs authentication or server-side access rules.

Free · no sign-up · runs in your browser

AI Crawler Policy Builder

Allow AI search, block AI training, or anything in between: pick a policy and copy the exact robots.txt lines, with what each line means according to the vendor that runs the crawler.

crawler tokens: 25
companies: 11
documentation reviewed: 2026-10-31

Your robots.txt lines

Block AI training, stay in AI search · 8 tokens blocked, 17 allowed

1# AI crawler rules: Block AI training, stay in AI search - vendor documentation reviewed 2026-10-31 (backlnk.com/tools/ai-crawler-policy)
2
3# AI training: opt out
4User-agent: GPTBot
5User-agent: ClaudeBot
6User-agent: Google-Extended
7User-agent: Applebot-Extended
8User-agent: meta-externalagent
9User-agent: Amazonbot
10User-agent: MistralAI-Training
11Disallow: /
12
13# Open web datasets: opt out
14User-agent: CCBot
15Disallow: /

Include a comment line with the choice and the review date

Paste the lines into the robots.txt at the root of your site (for example https://example.com/robots.txt), next to your existing groups.

Before you paste

Check:If your robots.txt already has a group that names one of these tokens, edit that group instead of adding a second one: crawlers merge every group that names them, and an existing "Allow: /" would win the tie against "Disallow: /".
Good to know:Allowed crawlers get no lines: they keep following your other rules. If your "*" group disallows everything, add an "Allow: /" group for the crawlers you want to keep.
Good to know:robots.txt is a request, not access control: it states what you ask crawlers to do. Use authentication or server rules to keep content private.

What these lines do

Per crawler token, with the effect its vendor documents. Documentation reviewed 2026-10-31.

What the generated robots.txt lines mean for each crawler token, with the vendor documentation they are based on
Crawler	Purpose	Your rule	What it means	Source
GPTBotOpenAI	AI training	Blocked	OpenAI: disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.	OpenAI docs(opens in a new tab)reviewed 2026-10-31
OAI-SearchBotOpenAI	AI search	Allowed	No line added: OAI-SearchBot keeps following your other robots.txt rules (its own group, or "*" when none names it).	OpenAI docs(opens in a new tab)reviewed 2026-10-31
ChatGPT-UserOpenAI	User-triggered fetch	Allowed	No line added: ChatGPT-User keeps following your other robots.txt rules (its own group, or "*" when none names it). The vendor documents that robots.txt rules may not apply to this user-triggered fetcher.	OpenAI docs(opens in a new tab)reviewed 2026-10-31
ClaudeBotAnthropic	AI training	Blocked	Anthropic: restricting ClaudeBot signals that the site's future materials should be excluded from AI model training datasets.	Anthropic docs(opens in a new tab)reviewed 2026-10-31
Claude-SearchBotAnthropic	AI search	Allowed	No line added: Claude-SearchBot keeps following your other robots.txt rules (its own group, or "*" when none names it).	Anthropic docs(opens in a new tab)reviewed 2026-10-31
Claude-UserAnthropic	User-triggered fetch	Allowed	No line added: Claude-User keeps following your other robots.txt rules (its own group, or "*" when none names it).	Anthropic docs(opens in a new tab)reviewed 2026-10-31
GooglebotGoogle	Search engine	Allowed	No line added: Googlebot keeps following your other robots.txt rules (its own group, or "*" when none names it).	Google docs(opens in a new tab)reviewed 2026-10-31
Google-ExtendedGoogle	AI training & grounding (control token)	Opted out	Google: Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal. Crawling itself is done with existing Google user agents. Control token: it does not crawl; it tells the vendor how content fetched by its other crawlers may be used.	Google docs(opens in a new tab)reviewed 2026-10-31
bingbotMicrosoft Bing	Search engine	Allowed	No line added: bingbot keeps following your other robots.txt rules (its own group, or "*" when none names it). Robots.txt support could not be re-verified from the vendor's documentation at review. Robots.txt support for bingbot could not be re-verified from a machine-readable Bing page at review.	Microsoft Bing docs(opens in a new tab)reviewed 2026-10-31
ApplebotApple	Search engine	Allowed	No line added: Applebot keeps following your other robots.txt rules (its own group, or "*" when none names it). Follows the Googlebot group when no group names it (vendor documentation).	Apple docs(opens in a new tab)reviewed 2026-10-31
Applebot-ExtendedApple	AI training (control token)	Opted out	Apple: pages that disallow Applebot-Extended can still be included in search results; Applebot-Extended only determines how data crawled by Applebot is used. Control token: it does not crawl; it tells the vendor how content fetched by its other crawlers may be used.	Apple docs(opens in a new tab)reviewed 2026-10-31
PerplexityBotPerplexity	AI search	Allowed	No line added: PerplexityBot keeps following your other robots.txt rules (its own group, or "*" when none names it).	Perplexity docs(opens in a new tab)reviewed 2026-10-31
Perplexity-UserPerplexity	User-triggered fetch	Allowed	No line added: Perplexity-User keeps following your other robots.txt rules (its own group, or "*" when none names it). The vendor documents that robots.txt rules may not apply to this user-triggered fetcher.	Perplexity docs(opens in a new tab)reviewed 2026-10-31
meta-externalagentMeta	AI training	Blocked	Meta: add a disallow for the crawler in robots.txt to block it; changes can take up to 24 hours.	Meta docs(opens in a new tab)reviewed 2026-10-31
meta-webindexerMeta	AI search	Allowed	No line added: meta-webindexer keeps following your other robots.txt rules (its own group, or "*" when none names it).	Meta docs(opens in a new tab)reviewed 2026-10-31
meta-externalfetcherMeta	User-triggered fetch	Allowed	No line added: meta-externalfetcher keeps following your other robots.txt rules (its own group, or "*" when none names it). The vendor documents that robots.txt rules may not apply to this user-triggered fetcher.	Meta docs(opens in a new tab)reviewed 2026-10-31
facebookexternalhitMeta	Link previews	Allowed	No line added: facebookexternalhit keeps following your other robots.txt rules (its own group, or "*" when none names it). The vendor documents that robots.txt rules may not apply to this user-triggered fetcher.	Meta docs(opens in a new tab)reviewed 2026-10-31
AmazonbotAmazon	Products & AI training	Blocked	Amazon: its crawlers honour robots.txt user-agent and allow/disallow directives and may use a cached copy from the last 30 days.	Amazon docs(opens in a new tab)reviewed 2026-10-31
Amzn-SearchBotAmazon	AI search	Allowed	No line added: Amzn-SearchBot keeps following your other robots.txt rules (its own group, or "" when none names it). Amazon documents that if robots.txt doesn't mention Amzn-SearchBot but allows other search bots, it follows those directives; which bots count is not specified, so Backlnk evaluates its own group or "" only.	Amazon docs(opens in a new tab)reviewed 2026-10-31
Amzn-UserAmazon	User-triggered fetch	Allowed	No line added: Amzn-User keeps following your other robots.txt rules (its own group, or "*" when none names it). The vendor documents that robots.txt rules may not apply to this user-triggered fetcher.	Amazon docs(opens in a new tab)reviewed 2026-10-31
CCBotCommon Crawl	Open web dataset	Blocked	Common Crawl: to prevent crawling, add "User-agent: CCBot / Disallow: /" to robots.txt. Common Crawl notes that other crawlers falsely identify as CCBot.	Common Crawl docs(opens in a new tab)reviewed 2026-10-31
DuckAssistBotDuckDuckGo	AI search	Allowed	No line added: DuckAssistBot keeps following your other robots.txt rules (its own group, or "*" when none names it).	DuckDuckGo docs(opens in a new tab)reviewed 2026-10-31
MistralAI-TrainingMistral AI	AI training	Blocked	Mistral: webmasters can disallow this user agent in robots.txt; it is not used for search indexing or live answers.	Mistral AI docs(opens in a new tab)reviewed 2026-10-31
MistralAI-IndexMistral AI	AI search	Allowed	No line added: MistralAI-Index keeps following your other robots.txt rules (its own group, or "*" when none names it).	Mistral AI docs(opens in a new tab)reviewed 2026-10-31
MistralAI-UserMistral AI	User-triggered fetch	Allowed	No line added: MistralAI-User keeps following your other robots.txt rules (its own group, or "*" when none names it).	Mistral AI docs(opens in a new tab)reviewed 2026-10-31

How it works

Lines that do exactly what you chose

The rules follow RFC 9309, the robots.txt standard, and the crawler list comes from each vendor's own documentation. Nothing is sent anywhere: the lines are built in your browser.

Rule 1
One group per purpose
Every token you block gets a "User-agent" line in a group that ends with "Disallow: /". Crawlers you allow get no lines and keep following the rest of your robots.txt.
Rule 2
Exact RFC 9309 matching
A crawler obeys only the groups that name its token (case-insensitive) and uses "*" only when none does. Groups for the same token are merged, the longest matching rule wins and Allow wins a tie.
Rule 3
Documented fallbacks handled
Applebot follows the Googlebot group when no group names it. If you block one but not the other, the builder adds an explicit "Allow: /" group.

What robots.txt can and cannot do

It can

Ask a named crawler to stay away from your whole site or from parts of it.
Separate AI training from AI search, because vendors document a separate token for each.
Opt out of uses through control tokens (Google-Extended and Applebot-Extended). They do not crawl; the vendor reads them to decide how content its other crawlers fetch may be used.

It cannot

Enforce anything. RFC 9309 says its rules are not a form of access authorization; private content needs authentication.
Bind every fetcher. Vendors document that ChatGPT-User, Perplexity-User, meta-externalfetcher, facebookexternalhit and Amzn-User may not follow robots.txt when a user triggers the fetch.
Fix a conflicting group elsewhere in the file. Groups for the same token are merged, so an existing “Allow: /” wins a tie with a new “Disallow: /”.

Sources reviewed

Every crawler comes from its vendor's documentation

Reviewed on 2026-10-31. Tokens that no official vendor page confirms are deliberately left out, so the builder never writes rules for guessed names.

OpenAI
GPTBot, OAI-SearchBot, ChatGPT-User
Vendor documentation for OpenAI (opens in a new tab)
Reviewed 2026-10-31
Anthropic
ClaudeBot, Claude-SearchBot, Claude-User
Vendor documentation for Anthropic (opens in a new tab)
Reviewed 2026-10-31
Google
Googlebot, Google-Extended
Vendor documentation for Google (opens in a new tab)
Reviewed 2026-10-31
Microsoft Bing
bingbot
Vendor documentation for Microsoft Bing (opens in a new tab)
Reviewed 2026-10-31
Bing's crawler help page (bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) renders client-side and could not be read at review; the bingbot meta controls are from Bing's webmaster blog.
Apple
Applebot, Applebot-Extended
Vendor documentation for Apple (opens in a new tab)
Reviewed 2026-10-31
Perplexity
PerplexityBot, Perplexity-User
Vendor documentation for Perplexity (opens in a new tab)
Reviewed 2026-10-31
Meta
meta-externalagent, meta-webindexer, meta-externalfetcher, facebookexternalhit
Vendor documentation for Meta (opens in a new tab)
Reviewed 2026-10-31
Amazon
Amazonbot, Amzn-SearchBot, Amzn-User
Vendor documentation for Amazon (opens in a new tab)
Reviewed 2026-10-31
Common Crawl
CCBot
Vendor documentation for Common Crawl (opens in a new tab)
Reviewed 2026-10-31
DuckDuckGo
DuckAssistBot
Vendor documentation for DuckDuckGo (opens in a new tab)
Reviewed 2026-10-31
Mistral AI
MistralAI-Training, MistralAI-Index, MistralAI-User
Vendor documentation for Mistral AI (opens in a new tab)
Reviewed 2026-10-31

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?: No. GPTBot and OAI-SearchBot are separate tokens, and OpenAI documents each one. Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
Does blocking Google-Extended affect Google Search?: No. Google-Extended is a control token, not a crawler. Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal. Crawling itself is done with existing Google user agents.
Why does the builder sometimes add an Allow group for Applebot?: Applebot respects robots.txt directives targeted at Applebot. If robots instructions don't mention Applebot but mention Googlebot, Applebot follows the Googlebot instructions. When you block Googlebot but keep Applebot, the builder adds a group that names Applebot with "Allow: /" so it does not inherit the block. Under RFC 9309 a crawler named by a group ignores the "*" group, so copy any "*" rules you still want it to follow into that group.
Do AI assistants follow robots.txt when a user asks them to open a page?: Not always. The vendors of ChatGPT-User, Perplexity-User, meta-externalfetcher, facebookexternalhit and Amzn-User document that these user-triggered fetchers may not follow robots.txt. Claude-User and MistralAI-User are documented without that exception.
My robots.txt already has a group for one of these crawlers. Can I just add the new lines?: Edit the existing group instead. Under RFC 9309 a crawler merges every group that names it, the longest matching rule wins, and an Allow rule wins a tie, so an existing "Allow: /" for the same token would override the new "Disallow: /".
Is robots.txt enough to stop AI from using my content?: It is a request, not a lock. RFC 9309 states that its rules are not a form of access authorization: crawlers that honour the protocol follow them, others may not. Content that must stay private needs authentication or server-side access rules.

Check your live robots.txt

After you publish the lines, the free AI Crawler Access Checker reads your real robots.txt and shows which crawler each group and line allows or blocks.

AI crawler access in the top 10,000 sitesOpen dataset: what the robots.txt files of the most popular domains ask of AI crawlers.Explore the research AI Crawler Access CheckerSee which AI and search crawlers your robots.txt and meta robots let in, rule by rule.Open the checker

AI Crawler Policy Builder

2. Allow or block by purpose

3. Fine-tune per crawler (optional)

OAOpenAI

AnAnthropic

GGoogle

BMicrosoft Bing

ApApple

PxPerplexity

MMeta

AmAmazon

CCCommon Crawl

DDDuckDuckGo

MiMistral AI

Your robots.txt lines

Before you paste

What these lines do

Lines that do exactly what you chose

One group per purpose

Exact RFC 9309 matching

Documented fallbacks handled

What robots.txt can and cannot do

It can

It cannot

Every crawler comes from its vendor's documentation

Frequently asked questions

Check your live robots.txt

OpenAI

Anthropic

Google

Microsoft Bing

Apple

Perplexity

Meta

Amazon

Common Crawl

DuckDuckGo

Mistral AI