How to Fix AI Crawlers Blocked in robots.txt
Your robots.txt contains rules that disallow one or more AI crawlers (like GPTBot, Claude-User, PerplexityBot, or OAI-SearchBot) from fetching your site. That keeps your content out of ChatGPT, Perplexity, and Claude answers. The fix is to edit robots.txt so the AI user-agents you want are allowed, while keeping genuinely sensitive paths protected.
What this means
Robots.txt is a plain-text file at the root of your domain (yoursite.com/robots.txt) that tells automated crawlers which paths they may fetch. Each block starts with a User-agent: line naming a bot, followed by Allow: and Disallow: rules.
This audit flagged that your robots.txt disallows crawlers behind AI answer engines. The patterns that trigger it are:
- An explicit block:
User-agent: GPTBotfollowed byDisallow: / - A named block for
ClaudeBot,Claude-User,PerplexityBot,Google-Extended,CCBot, orOAI-SearchBotwithDisallow: / - A catch-all
User-agent: *withDisallow: /that blocks every bot, AI crawlers included
These bots do different jobs, and the distinction matters. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) gather content to train models. Retrieval bots (OAI-SearchBot and ChatGPT-User for OpenAI, Claude-User and Claude-SearchBot for Anthropic, PerplexityBot for Perplexity) fetch your pages live when a user asks a question the AI needs current sources to answer. Blocking the retrieval bots is what most directly removes you from AI answers and citations. Note that AI crawlers honor robots.txt voluntarily, so this file states your policy rather than enforcing it at the server.
Why it matters
AI answer engines only cite and summarize content their crawlers are allowed to fetch. If OAI-SearchBot or Claude-User hits a Disallow: /, your pages are excluded from ChatGPT and Claude answers and their citations. The same applies to Perplexity via PerplexityBot. As more people start research inside these tools instead of a search box, a blanket AI block cuts you out of that discovery channel, and AI-referred visitors tend to arrive with high intent.
There is a classic-search angle too. A broad User-agent: * with Disallow: / does not only stop AI bots, it stops Googlebot from crawling at all, which quietly tanks organic rankings. Google's AI Overviews run on the same index Googlebot populates, so keeping Googlebot allowed is what makes you eligible for AI Overviews, with no separate opt-in. The fix is precision: unblock the AI and search crawlers you want, keep truly private paths disallowed, and drop any blanket block.
How to fix it
- 1
Read your current robots.txt
Open yoursite.com/robots.txt in a browser to see the live file. Look for any
Disallow: /lines and note whichUser-agentblock each sits under. A rule is scoped to the nearestUser-agentline above it, soUser-agent: GPTBotthenDisallow: /blocks only GPTBot, whileUser-agent: *thenDisallow: /blocks everything. Identify whether you have named AI-bot blocks, a global block, or both. - 2
Remove or narrow the blocking rules
For each AI or search crawler you want to allow, either delete its
Disallow: /block or replace/with the specific paths you actually want kept private, like/admin/or/cart/. If a catch-allUser-agent: *withDisallow: /is the culprit, change it toDisallow:(empty, meaning allow all) or scope it to real private directories. An emptyDisallow:and no rule at all both mean fully allowed. - 3
Name the AI crawlers you care about
You can let AI bots fall under
User-agent: *, or add explicit blocks to make your intent unambiguous. The tokens worth naming are GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI); ClaudeBot, Claude-User, and Claude-SearchBot (Anthropic); and PerplexityBot (Perplexity). Keep Googlebot allowed too, since it feeds both classic rankings and Google AI Overviews. If you want AI citations but not model training, allow the retrieval bots (OAI-SearchBot, Claude-User, PerplexityBot) and disallow the training bots (GPTBot, ClaudeBot, Google-Extended, CCBot). See the code example. - 4
Apply the change on your platform
On WordPress with Yoast or Rank Math, edit robots.txt through the plugin's file editor rather than fighting the virtual file WordPress generates. On Shopify, robots.txt is templated, so edit
robots.txt.liquidin your theme. On Wix and Squarespace, use the built-in robots.txt editor in SEO settings. On raw HTML, Next.js, or any custom stack, edit the physical/public/robots.txt(or your framework's route handler) and redeploy. The file must be reachable at the domain root, not in a subfolder. - 5
Validate and re-run the audit
Re-fetch yoursite.com/robots.txt and confirm the AI user-agents no longer carry
Disallow: /. Watch for ordering traps: within one user-agent block a more specificAllow:can override a broaderDisallow:, but rules across separate blocks do not interact. Run the audit again to confirm the flag clears. Changes take effect whenever each bot next re-reads your robots.txt, which they do on their own schedule (often within a day).
Example
# Allow all bots by default, block only private areas
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
# OpenAI (ChatGPT)
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Anthropic (Claude)
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
# Google-Extended controls Gemini training only; it is not a
# separate crawler. Keep Googlebot allowed for Search and AI Overviews.
User-agent: Google-Extended
Allow: /
# Point crawlers to your sitemap
Sitemap: https://www.yoursite.com/sitemap.xmlA robots.txt that welcomes AI answer-engine and search crawlers while keeping private paths blocked. Replace the example Disallow paths with your own.
Platform-specific steps
WordPress serves a virtual robots.txt by default. In Yoast, go to Yoast SEO > Tools > File editor to create and edit a real robots.txt. In Rank Math, go to Rank Math > General Settings > Edit robots.txt. Remove any Disallow: / under AI or * user-agents and save. Avoid editing at the server if a plugin is managing the file, or the two will conflict.
Shopify generates robots.txt from a template. In your admin, go to Online Store > Themes > Edit code, then create or open templates/robots.txt.liquid. Add or remove Disallow/Allow rules for the AI user-agents in Liquid, save, and the changes render at yourstore.com/robots.txt.
Both offer a built-in robots.txt editor in SEO settings (Wix: Marketing & SEO > SEO Tools > Robots.txt Editor; Squarespace exposes it under crawling/SEO settings). Edit the AI-crawler rules there. You cannot upload a physical file to the domain root on these hosts, so the built-in editor is the only supported path.
For a static file, edit /public/robots.txt and redeploy. For a dynamic route in Next.js App Router, edit app/robots.ts (or robots.js) so the returned rules allow the AI user-agents, then redeploy. Verify the file resolves at the exact domain root with a 200 status and Content-Type: text/plain.
Frequently asked
For most sites that want visibility, allow them. Being crawlable is what gets you cited and summarized in ChatGPT, Perplexity, and Claude. The main reason to block is content licensing: publishers who do not want their work used for model training block the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) while still allowing the retrieval bots (OAI-SearchBot, Claude-User, PerplexityBot) so they stay citable. If you are unsure, allow all of them, which maximizes reach.
Blocking the dedicated AI tokens (GPTBot, ClaudeBot, Google-Extended) does not affect classic Google rankings, because those are separate from Googlebot. But a broad User-agent: * with Disallow: / blocks Googlebot too, which does destroy rankings. Google-Extended only controls whether your content trains Gemini; it has no effect on Search or on whether you appear in AI Overviews, both of which depend on the regular Googlebot crawl.
Not instantly. Crawlers cache robots.txt and re-fetch it on their own schedules, so a change usually propagates within hours to a day as each bot re-reads the file. Robots.txt is also a voluntary standard. Most major bots honor it, but some have been documented ignoring it or masking their identity, so if you need hard enforcement, block by user-agent or IP at the server or CDN level instead of relying on robots.txt alone.
Both are OpenAI's, with different jobs. GPTBot is the training crawler that gathers content to train future models. OAI-SearchBot powers search and citations inside ChatGPT, and ChatGPT-User fetches a page when a user's question needs a real-time lookup. If your goal is to appear in ChatGPT's answers with a link back, the retrieval bots (OAI-SearchBot and ChatGPT-User) are the ones to keep allowed. OpenAI supports controlling each independently.
At the root of your domain, reachable at exactly yoursite.com/robots.txt. It cannot sit in a subfolder, and it only covers its own host. A subdomain like blog.yoursite.com needs its own separate robots.txt. Confirm the file returns a 200 status as plain text, not a redirect or an HTML error page.
Does your site have this issue?
Run a free, AI-powered audit and we’ll flag this and 150+ other checks in about a minute. No signup.