How to Block AI Crawlers with robots.txt Without Leaving AI Search
AI companies run separate bots for training, AI search and user requests. Here are the documented tokens, how robots.txt matching works and copy-ready configurations.
Topic
How AI assistants and AI search summaries find, read, quote and cite websites, and what site owners control.
5 articles
AI companies run separate bots for training, AI search and user requests. Here are the documented tokens, how robots.txt matching works and copy-ready configurations.
A plain explanation of the llms.txt proposal: the format with a full example, how it differs from robots.txt and sitemaps, and what vendors really document about reading it.
Google says AI Overviews need no special schema. Here is what structured data still does, how to mark up your organization and authors in JSON-LD, and how to validate it.
No method guarantees an AI citation. Here is what AI search providers actually document, what you control, and how to write self-contained passages that serve readers and quote cleanly.
Original BACKLNK research: we evaluated 25 crawler tokens against the robots.txt answers of 5,996 reachable top-10,000 sites to see who blocks AI training, AI search and user fetchers.