Fix "Robots.txt Not Accessible" (+ Add Sitemap)
Your audit flagged that your robots.txt either can't be fetched cleanly (it returns a server error, times out, or serves the wrong content type) or it loads but has no Sitemap directive. The unreachable case is the urgent one: when Google gets a 5xx or timeout on robots.txt, it stops crawling your site until the file is reachable again, and a long outage can cause pages to drop from the index. A missing Sitemap line is lower-stakes, but it means crawlers and AI bots have to find your sitemap on their own instead of being pointed straight to it. Both fixes are quick: serve a valid robots.txt at your root with an HTTP 200 response, and add one absolute-URL Sitemap line.
What this means
Every crawler (Googlebot, Bingbot, and AI bots like GPTBot, ClaudeBot, and PerplexityBot) requests https://yourdomain.com/robots.txt before crawling anything else. This audit fires when that request doesn't return a clean, usable response.
Two distinct problems are bundled under this issue:
robots_not_accessible — The file can't be fetched properly. Common causes: the server returns a 5xx (500/502/503) error, the request times out, the file redirects in a loop, it's served with the wrong content type (text/html instead of text/plain), or a firewall/WAF blocks the crawler from reaching it. Note the key distinction: a plain 404 (file doesn't exist) is treated by Google as "no rules, crawl everything" and is technically fine. A 5xx or timeout is read as "unreachable," which is the dangerous state.
robots_missing_sitemap — The file loads fine but has no Sitemap: line. That line tells every crawler exactly where your XML sitemap lives. Without it, bots fall back to guessing (/sitemap.xml) or rely only on what you submitted in Search Console, and AI crawlers that don't use Search Console lose an easy pointer to your full URL list.
Why it matters
The unreachable case is genuinely risky. When Googlebot requests robots.txt and gets a 5xx or a timeout, it doesn't assume it's safe to crawl. To avoid hitting URLs you may have meant to block, Google pauses crawling and falls back to its last cached copy of robots.txt (normally cached about 24 hours, held longer during errors). Per Google's own documentation, if the error persists for about 30 days Google starts treating the site as if robots.txt doesn't exist when the site is otherwise reachable, but if the whole site looks unreachable it stops crawling, and previously indexed URLs can gradually drop out. This is one of the few technical-SEO issues that can quietly deindex an otherwise healthy site, so a robots.txt that reliably returns 200 is non-negotiable.
The missing-sitemap case is lower severity but still worth fixing. The Sitemap: directive is the most reliable, user-agent-independent way to hand every crawler your canonical URL list. It helps Google and Bing find new and updated pages faster, and it matters for AI answer engines too: crawlers like GPTBot, ClaudeBot, and PerplexityBot have no access to your Search Console sitemap submissions, so the robots.txt line is often the only machine-readable pointer they get. If you want your content discoverable and citable by AI Overviews, ChatGPT, and Perplexity, make it trivial for their bots to find your full sitemap.
How to fix it
- 1
Confirm the exact failure first
Open the file at https://yourdomain.com/robots.txt in a browser and check two things: does it load, and does it return HTTP 200? Use curl to see the real status code and headers:
curl -I https://yourdomain.com/robots.txt. You want a 200 withContent-Type: text/plain. A 500/502/503, a timeout, a redirect chain, orContent-Type: text/htmlall trip this audit. Also test with a crawler user-agent (curl -A "Googlebot" https://yourdomain.com/robots.txt), because a firewall or WAF sometimes serves errors to bots while returning 200 to your browser. - 2
Serve a valid robots.txt that returns 200
The file must live at the root of each hostname (protocol + host +
/robots.txt) and respond with a 200 status and plain-text content. If it's currently returning 5xx, the fix is server-side: the file may be generated by an app that's throwing errors, or the path may be misconfigured. For a static file, place a plainrobots.txtin your web root. A safe, permissive baseline that allows all crawling is:User-agent: *
Disallow:
An empty
Disallow:means "block nothing." Never leave a strayDisallow: /unless you truly intend to block your whole site. - 3
Add an absolute-URL Sitemap line
Append a
Sitemap:directive pointing to your XML sitemap. It must be a fully qualified absolute URL including the protocol and host; a relative path likeSitemap: /sitemap.xmlis silently ignored. The directive is independent of anyUser-agentblock, so it can go anywhere in the file (conventionally at the bottom). If you have several sitemaps, list each on its own line or point to a single sitemap index. Example:Sitemap: https://yourdomain.com/sitemap.xml
- 4
Fix it on your specific platform
On WordPress, don't hand-edit if you run Yoast or Rank Math; both serve a virtual robots.txt and inject the Sitemap line automatically (Yoast: Tools > File editor; Rank Math: General Settings > Edit robots.txt). Make sure no leftover physical robots.txt in your web root is overriding the plugin's virtual one. On Shopify, robots.txt is auto-generated and already includes the sitemap, so if it's unreachable the cause is usually a DNS or CDN issue rather than the file. On Next.js, add an
app/robots.tsroute (orpages/api/robots) and confirm it returns 200 andtext/plain. Behind Cloudflare, check that a firewall rule, Bot Fight Mode, or 'Under Attack' mode isn't serving a challenge page (which reads as non-200) to bots requesting robots.txt. - 5
Re-test and validate
After deploying, re-run curl to confirm a clean 200 and plain-text content type. Open Search Console's robots.txt report (Settings > robots.txt) to see how Google last fetched the file and whether it parsed the Sitemap line. Then submit your sitemap under Sitemaps in Search Console so Google has a direct record independent of robots.txt. Finally, re-run your site audit to clear both the robots_not_accessible and robots_missing_sitemap flags.
Example
# robots.txt - served at https://yourdomain.com/robots.txt (HTTP 200, text/plain)
User-agent: *
Disallow:
# Optional: keep AI crawlers explicitly welcome (they are allowed by default anyway)
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Sitemap must be an absolute URL; relative paths are ignored
Sitemap: https://yourdomain.com/sitemap.xmlA complete, valid robots.txt: allows all crawling, blocks nothing sensitive, and points every crawler to the sitemap with an absolute URL.
Frequently asked
Not for crawling. Google treats a missing robots.txt (a clean 404) as 'no rules, crawl everything,' which is fine. The dangerous responses are 5xx errors and timeouts, which Google reads as 'unreachable' and responds to by pausing crawl and falling back to a cached copy. If this audit flagged you, it's usually because the file errors out, or it exists but lacks a Sitemap line. Creating a proper robots.txt fixes both and gives you control you don't have when the file is simply absent.
When Googlebot can't fetch robots.txt (5xx or timeout), it can't know which URLs you disallowed, so rather than risk crawling blocked pages it pauses and uses the last cached copy. Per Google's documentation, if the failure lasts around 30 days and the whole site looks unreachable, Google stops crawling and previously indexed URLs can gradually drop out. A robots.txt that reliably returns HTTP 200 prevents this.
You don't have to, but you should. Search Console submission only covers Google. The Sitemap line in robots.txt is read by every crawler, including Bing and AI bots like GPTBot, ClaudeBot, and PerplexityBot that have no access to your Search Console. It's the one place that makes your sitemap discoverable to all of them at once, so keep both.
No. The Sitemap directive requires a fully qualified absolute URL including the protocol and host, like https://yourdomain.com/sitemap.xml. A relative path such as /sitemap.xml is silently ignored. If your sitemap line isn't being picked up, a relative path is the most common reason.
At the root of each hostname: https://yourdomain.com/robots.txt. It applies only to the exact protocol, host, and port it's served from, so https:// and a www. subdomain each need their own reachable file (usually handled automatically if both resolve to the same server). It won't work in a subfolder like /blog/robots.txt.
Does your site have this issue?
Run a free, AI-powered audit and we’ll flag this and 150+ other checks in about a minute. No signup.