Fix Googlebot Blocked in robots.txt (Critical)
Your robots.txt contains a rule that tells Googlebot (or every crawler) not to fetch your site. Until you remove or narrow that Disallow, Google can't read your pages, and they eventually drop out of Search. It's one of the fastest fixes on any audit: usually one or two lines in a single file.
What this means
This flag means your robots.txt file, served at https://yourdomain.com/robots.txt, contains a directive that blocks Google's crawler. It happens two ways: a rule aimed at Google specifically, or a blanket rule aimed at every bot.
The classic mistake looks like this:
User-agent: *
Disallow: /User-agent: * applies to all crawlers, and Disallow: / blocks the entire site starting at the root. A Google-specific version does the same thing with User-agent: Googlebot.
robots.txt is a crawl-control file, not an access-control file. A Disallow tells compliant crawlers not to fetch those URLs. It requires no login and hides nothing from people. Googlebot obeys it, so a site-wide Disallow: / means Google stops requesting your pages.
The most common cause is a leftover staging rule. Many CMS platforms and dev environments ship a "discourage search engines" setting that writes Disallow: / to keep a work-in-progress site out of Google. If that setting survives the move to production, your live site launches blocking its own crawler.
Why it matters
Crawling is the first step in Google's pipeline: crawl, render, index, rank. If Googlebot can't crawl, everything downstream stalls. Pages that were already indexed gradually drop out as Google recrawls and hits the block, and new or updated pages never get picked up. On a live site this is a traffic emergency, not a minor warning.
One trap is worth knowing. A URL blocked in robots.txt can still appear in Search as a bare, description-less link if other sites point to it, because Google indexed the URL without being allowed to read the page. It can't see your title, content, canonical, or structured data. So a block rarely removes a URL cleanly; it just strips out everything that makes the result useful or rank-worthy. If your real goal is to keep a page out of the index, use a noindex tag (which needs the page to be crawlable), not a robots.txt Disallow.
The answer-engine angle makes this more urgent. Google's AI Overviews are built on the same crawl-and-index foundation, so a Googlebot block also weakens your eligibility to be cited there. Separately, most AI answer engines respect robots.txt under their own user-agents: GPTBot and OAI-SearchBot (OpenAI), PerplexityBot, ClaudeBot, Google-Extended (Gemini training), and others. A sloppy User-agent: * / Disallow: / blocks all of them at once, so your content goes invisible to both classic search and the AI systems that increasingly send referral traffic. Fixing the block restores visibility across both channels.
How to fix it
- 1
Read your live robots.txt and find the blocking rule
Open
https://yourdomain.com/robots.txtin a browser, orcurlit, and read it top to bottom. Look for any group whoseUser-agentline is*orGooglebotfollowed byDisallow: /(a bare slash blocks the whole site). Remember how matching works: Google applies the single most specific group, so if aGooglebotgroup exists it overrides the*group for Googlebot, and the two are never combined. Note the exact lines to change before editing anything. - 2
Remove or narrow the Disallow so the site is crawlable
The goal is to allow crawling of everything you want indexed. A safe, permissive robots.txt for most sites is
User-agent: *thenDisallow:with an empty value, which blocks nothing, plus aSitemap:line. If you only meant to hide specific areas like an admin path or internal search, replaceDisallow: /with targeted paths such asDisallow: /wp-admin/orDisallow: /cart. Never leave a bareDisallow: /on a production site you want in Google. - 3
Fix it at the platform that generates the file
On WordPress, uncheck Settings > Reading > 'Discourage search engines from indexing this site'; that box makes WordPress serve a blocking virtual robots.txt. With Yoast or Rank Math, edit robots.txt directly in the plugin's file editor. On Shopify, edit
robots.txt.liquidunder Online Store > Themes > Edit code. On Wix, turn off 'Block search engines' in SEO settings. On Squarespace robots.txt is managed for you, so a block usually comes from a page-level 'Hide from search engines' setting instead. For raw HTML, Next.js, or a static host, edit the physical/robots.txt(orapp/robots.tsin Next.js) and redeploy. - 4
Check for a CDN or edge rule rewriting the file
If robots.txt looks correct in your CMS but the live URL still shows a block, something at the edge may be overriding it. On Cloudflare, look for a Transform Rule, Redirect Rule, or Worker that intercepts
/robots.txt. A WAF or bot-protection rule can also serve Googlebot a challenge page that reads as a block. Confirm you're editing the file that actually gets served by comparing the CMS version against the live fetched version. - 5
Validate with Google's own tools
In Google Search Console, open the robots.txt report under Settings to confirm Google fetched the updated file with no errors. Then run a few important URLs through the URL Inspection tool and check that 'Crawl allowed?' says Yes. Google recrawls robots.txt roughly daily; once crawling is unblocked, use URL Inspection > Request Indexing on your key pages to speed things up.
- 6
Confirm you didn't just move the block to a meta tag
After unblocking robots.txt, verify the pages you want indexed don't carry
<meta name="robots" content="noindex">or anX-Robots-Tag: noindexHTTP header. Those are a separate block that only becomes visible once crawling is allowed. Check the page source and the response headers. If a page must stay out of the index, keep it crawlable and usenoindex, so Google can actually read and honor the directive.
Example
# Allow all standard crawlers (Googlebot, Bingbot, etc.)
User-agent: *
Disallow:
# Example: block only real private areas, not the whole site
# Disallow: /cart
# Disallow: /wp-admin/
# Allow: /wp-admin/admin-ajax.php
# Optional: opt out of AI training / answer engines by name
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
# Point crawlers to your sitemap (use your real URL)
Sitemap: https://yourdomain.com/sitemap.xmlA safe production robots.txt: crawlable for search, with AI training and answer bots opted out selectively. Adjust the AI groups to your preference.
Platform-specific steps
First uncheck Settings > Reading > 'Discourage search engines from indexing this site'; this setting makes WordPress serve a blocking virtual robots.txt. Then edit the file directly in Yoast (Yoast SEO > Tools > File editor) or Rank Math (Rank Math > General Settings > Edit robots.txt). Remove any Disallow: / and keep only targeted paths like Disallow: /wp-admin/.
Shopify generates robots.txt from a template you can override. Go to Online Store > Themes > Edit code and open (or create) robots.txt.liquid. Remove or narrow any rule producing a site-wide Disallow: /, save, then confirm the live /robots.txt reflects the change.
Open SEO settings and make sure 'Let search engines index your site' is enabled (or 'Block search engines from indexing' is off). Wix manages robots.txt for you, so the block is almost always this toggle rather than a hand-edited file.
Squarespace auto-manages robots.txt and won't let you site-block Google there, so this flag usually points to a page or site-level 'Hide from search engines' setting under SEO. Check site visibility and per-page settings, and remove the hide option on pages you want indexed.
Edit the physical public/robots.txt or the dynamic app/robots.ts (or pages/robots.txt) so it returns a permissive rule plus your sitemap URL, then redeploy. Fetch the live URL after deploy to confirm the CDN is serving the new file and not a cached copy.
If the CMS file is correct but the live URL still blocks, check the edge. Look for a Transform Rule, Redirect Rule, or Worker intercepting /robots.txt, and check whether a WAF or bot-fight rule is serving Googlebot a challenge page. Allow verified Googlebot and let the origin's real robots.txt pass through.
Frequently asked
No, but it removes the blocker. Google has to recrawl your robots.txt (usually within a day), then recrawl and reindex the affected pages, which can take days to a few weeks depending on your crawl budget. Speed it up by requesting a robots.txt refetch in Search Console and using URL Inspection > Request Indexing on your most important pages.
robots.txt controls crawling: don't fetch this URL. noindex controls indexing: you can read this, but don't put it in Search. They aren't interchangeable. If you Disallow a page in robots.txt, Google can't crawl it, so it never sees your noindex tag, and the URL can still surface as a bare link. To reliably keep a page out of the index, allow crawling and add noindex.
Target the AI crawlers by their specific user-agents rather than using a blanket rule. Add groups for GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, and Google-Extended with Disallow: /, while keeping User-agent: * (or a specific Googlebot group) permissive. Note that Google-Extended only controls Gemini training and grounding; it does not block regular Googlebot or your eligibility for AI Overviews, which rely on the standard index.
You're probably editing a different file than the one served. Fetch https://yourdomain.com/robots.txt directly and compare. Common causes: a CDN or Cloudflare rule rewriting the response, a WAF or bot-protection challenge served to Googlebot, an old cached version, or a staging config leaking to production. The live fetched file is the only one that counts.
It must sit at the exact root of the host: https://yourdomain.com/robots.txt. A file at /blog/robots.txt or anywhere else is ignored. Each host and protocol combination needs its own robots.txt, so blog.yourdomain.com and www.yourdomain.com are governed by separate files.
Does your site have this issue?
Run a free, AI-powered audit and we’ll flag this and 150+ other checks in about a minute. No signup.