How to Fix AI Crawlers Blocked in robots.txt

Robots.txt is a plain-text file at the root of your domain (yoursite.com/robots.txt) that tells automated crawlers which paths they may fetch. Each block starts with a User-agent: line naming a bot, followed by Allow: and Disallow: rules.

This critical fires for one group of bots only: the crawlers AI answers are drawn from. Those are OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, bingbot, Applebot, DuckAssistBot, Amzn-SearchBot, Meta-WebIndexer and MistralAI-Index. Most of them build an index an engine answers from; DuckAssistBot is different - DuckDuckGo says it "crawls pages in real-time for our AI-assisted answers, which prominently cite their sources", and documents no index of its own (DuckDuckGo help pages, read 2026-09-23). Either way, a block means the engine cannot reach your pages. The patterns that trigger it are:

• A named block, such as User-agent: PerplexityBot followed by Disallow: /
• A catch-all User-agent: * with Disallow: /, which applies to each of those bots that has no group of its own in your file
• A rule that blocks everything without saying Disallow: / literally, such as Disallow: /*

We read your file with the same longest-match parser the crawl itself is gated on, and we only call a bot blocked when your homepage is disallowed for it, or when every page we sampled across two or more sections of your site is. A single disallowed folder is never reported as a site-wide block.

Three other kinds of block do not raise this critical, because they do not cost you citations. Each gets its own separate notice on your report instead:

• Training crawlers - GPTBot, ClaudeBot, CCBot, Amazonbot, Applebot-Extended, meta-externalagent, MistralAI-Training, Bytespider. They collect content for model training. Blocking them is a normal content-licensing choice and does not remove you from AI answers.
• Fetchers that run when a person asks - ChatGPT-User, Perplexity-User, Claude-User, Meta-ExternalFetcher, Amzn-User, Google-Agent, MistralAI-User. These load a single page because someone asked an assistant about it. OpenAI, Perplexity, Meta, Amazon and Google all state that robots.txt may not be enforced for their user-triggered fetchers, so a block here may not even hold.
• Google-Extended - it controls Gemini training and grounding in Gemini apps, and has no crawler of its own. Google states it does not impact a site's inclusion in Google Search and is not used as a ranking signal (Google crawler documentation, updated 2026-07-14).

That distinction matters most when something else manages your robots.txt for you. Cloudflare's managed robots.txt adds Disallow: / for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent (Cloudflare documentation, updated 2026-08-03). Every one of those is a training or opt-out token, so that switch on its own does not raise this critical.

One limit worth stating plainly: robots.txt is a voluntary standard. It records your policy, it does not enforce anything at your server. If you need a block enforced, do it at your firewall or CDN.

AI answers link to pages an engine's crawler has already fetched. The crawler named in your report is how your pages reach that engine, and each operator says so in its own documentation: OpenAI recommends allowing OAI-SearchBot if you want to appear in ChatGPT search results; Perplexity and Anthropic run PerplexityBot and Claude-SearchBot for the same job; Microsoft reports where publisher content shows up across Copilot, AI-generated summaries in Bing and partner integrations (Bing Webmaster Tools, 2026-02-10); Apple's Applebot powers Spotlight, Siri and Safari search. A Disallow: / aimed at one of those user-agents is a standing instruction to stay out of that engine's answers.

Being fetchable is a precondition, not a promise. Allowing these crawlers does not guarantee you will be cited; nothing does. What a block does guarantee is that the engine cannot fetch your pages, so it has nothing of yours to surface.

There is a classic-search angle whenever the cause is a catch-all rule. A User-agent: * with Disallow: / does not only stop AI crawlers, it stops Googlebot, which quietly removes you from ordinary search results too. Google's AI Overviews and AI Mode are built on the same index Googlebot populates, so that one rule cuts both channels at once. There is no separate AI Overviews crawler to allow: whether your pages may appear there is a Search Console setting (Settings > Search generative AI) that no crawler can see, so no audit, ours included, can check it for you.

What this issue is not: a verdict on AI training. If you disallow GPTBot, ClaudeBot, CCBot or Google-Extended deliberately, that is a licensing decision, it is reported separately as a notice, and it is not what raised this critical.