Free Robots.txt Tester
Fetch any site’s robots.txt, review its crawl rules and sitemaps, and test whether a specific URL path is allowed or blocked for search engines.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search? Allow: /admin/help/ Sitemap: https://example.com/sitemap.xml
What is robots.txt?
robots.txt is a plain-text file at the root of your domain that tells search engine crawlers which parts of your site they may or may not crawl. A misconfigured robots.txt is one of the most common — and most damaging — technical SEO mistakes: a single stray Disallow: / can deindex an entire website.
What to check
- Make sure you’re not accidentally blocking important pages or your whole site.
- Declare your XML sitemap with a
Sitemap:directive. - Remember: robots.txt controls crawling, not indexing — use a
noindexmeta tag to keep a page out of search results.
Use the path tester above to confirm a specific URL is crawlable before you publish.
Frequently asked questions
Is the robots.txt tester free to use?
Yes, it is completely free with no signup. Fetch a site's robots.txt and check whether a path is allowed or blocked for a given user-agent as many times as you need.
Why is a page still showing in Google if it's blocked in robots.txt?
Robots.txt only stops crawling, not indexing, so a blocked URL can still appear in results if Google finds links to it elsewhere. To keep a page out of the index, allow crawling and add a noindex meta tag or X-Robots-Tag header, or restrict it with authentication.
Where should the robots.txt file be located?
It must sit at the root of each host, such as https://example.com/robots.txt, and applies only to that exact domain, subdomain, and protocol. A file in a subfolder is ignored, and subdomains like blog.example.com need their own robots.txt.
Embed this free tool on your site
Add this robots.txt tester to your own blog or website — it stays free for your visitors, and the short credit line links back here. Just copy the snippet and paste it into your page’s HTML.
<iframe src="https://seoaiaudits.com/embed/robots-txt-tester" width="100%" height="700" style="border:1px solid #e5e7eb;border-radius:12px;max-width:760px;width:100%" loading="lazy" title="Robots.txt Tester"></iframe> <p style="font-size:13px;font-family:system-ui,sans-serif;margin-top:8px;color:#6b7280"> Free <a href="https://seoaiaudits.com/tools/robots-txt-tester" target="_blank" rel="noopener">Robots.txt Tester</a> by <a href="https://seoaiaudits.com" target="_blank" rel="noopener">SEO AI Audits</a> </p>
Tip: adjust height if your layout needs more or less room. Please keep the credit line so others can find the tool.
Related free tools
Instant SEO Snapshot
One-click SEO & AI-readiness grade for any page — HTTPS, indexability, title, meta, headings, content & more. Embeddable on your own site.
AI Bot Access Checker
Can ChatGPT, Perplexity, Claude & Gemini crawl your site? Checks robots.txt + live CDN/WAF blocks.
AI Visibility Checker
See what ChatGPT, Claude & Perplexity know about your brand — and how to improve it.
NLP Content Analyzer
Topics, entities, search intent & content gaps an NLP model sees in your page.
AI Retrieval Checker
AI/AIO readiness — can ChatGPT, Perplexity & AI Overviews find, chunk & cite your page?
AI Crawler Log Analyzer
Upload your access log to see which AI bots (GPTBot, ClaudeBot, PerplexityBot) actually crawled you — with spoofed-vs-verified detection.
Want the full picture?
This tool checks one thing. Run a complete, free SEO audit across 26 factors in about a minute.
Run a free SEO audit