Skip to content
Backlnk
Free audit

AI Crawler Access Checker

See which AI and search crawlers your robots.txt and meta robots let in — rule by rule.

We read robots.txt for this host and the page's own robots directives.

Options
Also check this page with

Free, no sign-up. Every audit is saved privately to this browser — find it later in your history.

Your saved reports

What this tool does

Evaluates your robots.txt for every major AI and search crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bingbot and more) the way RFC 9309 defines it, shows the exact group and line that decides, and adds page-level meta robots and X-Robots-Tag directives.

What you get

  • Allowed / blocked matrix for each crawler, grouped by company and purpose
  • The exact robots.txt group and line behind every decision
  • Page-level noindex, nosnippet and bot-specific directives

Scope

robots.txt of the host plus one page's meta robots and X-Robots-Tag.

Pages are fetched politely by BacklnkBot, robots.txt is respected, and only public websites can be audited. Every finding shows the evidence it is based on and when it was measured.

Limitations

  • • robots.txt is a request, not an access control: it shows what you ask crawlers to do, not what they do.
  • • Some user-triggered fetchers (for example when a person asks an assistant to open a page) are documented by their vendors as not following robots.txt.
  • • Crawler names and purposes follow each vendor's documentation as reviewed on the date shown in the report.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?
Not necessarily. Vendors run separate crawlers for training, search and user-requested fetches, each with its own robots.txt token. The report lists them separately with each vendor's documented purpose.
Is Google-Extended a crawler?
No. It is a robots.txt token Google reads to decide whether content may be used for its generative AI models; crawling itself is done by Googlebot.

Guides

Learn the topic behind the ai crawler access check

More free tools

Learn more: AI Visibility guides · Technical SEO guides