AI Bot Access Checker

See exactly what your robots.txt says to each AI crawler — the search crawlers behind ChatGPT, Perplexity, Claude, Copilot and Meta AI, the fetchers that load a page when someone asks an assistant, and the training tokens. We read your public robots.txt with the same longest-match rules search engines use. We never send a request pretending to be someone else's crawler.

Example resultA sample report for example.com — run the tool on your own URL above for live results.
Search crawlers allowed
8 / 9
AI tokens blocked
6 of 26
Training tokens blocked
5 of 9
  • OAI-SearchBot
    OpenAI · indexes pages for ChatGPT search results
    Allowed in robots.txt
  • Claude-SearchBot
    Anthropic · indexes pages for Claude’s search results
    Allowed in robots.txt
  • PerplexityBot
    Perplexity · indexes pages so Perplexity can link them in answers
    Blocked in robots.txt
  • GPTBot
    OpenAI · AI training crawler
    Blocked in robots.txt
  • Google-Extended
    Google · Gemini training & Gemini-app grounding · opt-out token only
    Blocked in robots.txt

Why AI bot access matters

To be surfaced by ChatGPT search, Perplexity, Claude or Microsoft Copilot, their crawlers have to be able to fetch your pages. Plenty of sites block them without meaning to, usually because one Disallow line covers more crawlers than the owner realised. This tool reads your robots.txt the way the engines do and tells you, per crawler, what your own rules say.

What this checker reads — and what it cannot see

  • Per-bot robots.txt evaluation — the actual allow/deny rule each crawler sees, using Google’s longest-match semantics. We evaluate your homepage, and the deeper URL too if you paste one, because robots.txt rules are matched per path. We also run your rules against a few ordinary paths such as /about so that one disallowed folder is not mistaken for a site-wide block. Those extra paths are only measured against your rules on our side — nothing on your site is requested except robots.txt itself.
  • Grouped by what a block costs — search crawlers, user-triggered fetchers, training tokens, Google-Extended and classic search are five different decisions, not one.
  • Your Content-Signal lines, if you publish any — shown as a statement of preference, never scored.
  • Not your firewall. A CDN, WAF or bot-management rule can block a crawler no matter what robots.txt says, and those settings are private. We do not send requests wearing other companies’ bot names to probe for that: it is inaccurate on exactly the Cloudflare sites where it matters, and it conflicts with identifying ourselves honestly. Your server logs, or Cloudflare’s free AI Crawl Control, are where the truth about edge blocks lives.

Which tokens we check

OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (Claude-SearchBot, Claude-User, ClaudeBot, anthropic-ai), Perplexity (PerplexityBot, Perplexity-User), Google (Googlebot, Google-Agent, Google-Extended), Microsoft Bing, Apple (Applebot, Applebot-Extended), Amazon (Amzn-SearchBot, Amzn-User, Amazonbot), Meta (Meta-WebIndexer, Meta-ExternalFetcher, meta-externalagent), Mistral (MistralAI-Index, MistralAI-User, MistralAI-Training), DuckDuckGo (DuckAssistBot), Common Crawl (CCBot) and ByteDance (Bytespider).

Blocking training crawlers (GPTBot, CCBot, Bytespider, Amazonbot) is a legitimate licensing choice and does not remove you from AI answers. Blocking search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot) is what keeps you out of them.

Frequently asked questions

Does this tool test whether my firewall blocks AI bots?

No, and it deliberately does not try. Testing that would mean sending requests carrying another company's crawler name from our servers, which is not honest self-identification and is unreliable anyway: Cloudflare's fake-bot rules block those requests by design, so on a Cloudflare site the result would show blocks that do not exist. What this tool reports is your robots.txt, read accurately. To see real edge blocks, check your server logs with our AI Crawler Log Analyzer, or open Cloudflare's AI Crawl Control, which is free on all plans and shows which AI crawlers reached your site.

Does blocking GPTBot stop ChatGPT from citing me?

No. GPTBot is OpenAI's training crawler; the crawler that builds the index behind ChatGPT search is OAI-SearchBot. You can block GPTBot to keep your content out of model training while leaving OAI-SearchBot allowed so your pages can still be surfaced and linked. The same split exists at Anthropic (ClaudeBot trains, Claude-SearchBot indexes) and Amazon (Amazonbot versus Amzn-SearchBot).

What does blocking Google-Extended actually do?

Google-Extended controls whether your content is used to train Gemini models and to ground answers in Gemini apps. Google's own documentation states it does not impact a site's inclusion in Google Search and is not a ranking signal, so blocking it does not remove you from AI Overviews or AI Mode. Whether your pages may appear in those is a separate Search Console setting, under Settings then Search generative AI, that no outside crawler can see. Google-Extended also has no crawler of its own, so it never appears in your access logs.

Why do some bots say “Couldn’t tell”?

That appears when your robots.txt returned a server error or did not answer at all. A 4xx response means there are no rules and nothing is disallowed, which we report as such, but a 5xx or an unreachable file is not evidence in either direction — RFC 9309 and Google's own spec treat it as a reason to stop crawling, not as permission. Rather than guess, we say we could not tell, and you should fix the robots.txt response first.

Why is a blocked ChatGPT-User or Perplexity-User only a note?

Those are user-triggered fetchers: they load a page because a person asked an assistant about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt, and Meta, Amazon and Google say the same of theirs. So a robots.txt block on those is a stated preference rather than a guarantee, and we label it that way. Anthropic is the exception: it says all its bots, including Claude-User, honour robots.txt.

Related free tools

Instant SEO Snapshot

One-click SEO & AI-readiness grade for any page — HTTPS, indexability, title, meta, headings, content & more. Embeddable on your own site.

AI Model Recall Checker

Ask free AI models the unbranded questions your buyers type, and count how many name your brand unprompted — with each model and its training cutoff shown. It measures what those models remember from training, not a live check of ChatGPT, Claude or Perplexity; a built-in panel lets you spot-check the real engines by hand.

NLP Content Analyzer

Topics, entities, search intent & content gaps an NLP model sees in your page.

AI Retrieval Checker

AI-retrieval readiness for one page — an AI model’s read of how findable and quotable it is, with fixes.

AI Crawler Log Analyzer

Upload your access log to see which AI bots (GPTBot, ClaudeBot, PerplexityBot) actually crawled you — with IP verification where the operator publishes ranges. Runs in your browser.

AI Citation Report (Bing Copilot)

Upload your Bing Webmaster AI Performance export to see which of your pages Microsoft Copilot and Bing’s AI answers cite, and how that trends. Covers Copilot and Bing only — Google’s AI numbers are in Search Console’s own Generative AI report. Runs in your browser.

Want the full picture?

This tool checks one thing. Run a complete, free SEO audit across 28 modules.

Run a free SEO audit