How to Block AI Crawlers with robots.txt Without Leaving AI Search
AI companies run separate bots for training, AI search and user requests. Here are the documented tokens, how robots.txt matching works and copy-ready configurations.
See which AI and search crawlers your robots.txt and meta robots let in — rule by rule.
Evaluates your robots.txt for every major AI and search crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bingbot and more) the way RFC 9309 defines it, shows the exact group and line that decides, and adds page-level meta robots and X-Robots-Tag directives.
robots.txt of the host plus one page's meta robots and X-Robots-Tag.
Pages are fetched politely by BacklnkBot, robots.txt is respected, and only public websites can be audited. Every finding shows the evidence it is based on and when it was measured.
Guides
AI companies run separate bots for training, AI search and user requests. Here are the documented tokens, how robots.txt matching works and copy-ready configurations.
Original BACKLNK research: we evaluated 25 crawler tokens against the robots.txt answers of 5,996 reachable top-10,000 sites to see who blocks AI training, AI search and user fetchers.
Learn more: AI Visibility guides · Technical SEO guides