Free Robots.txt Tester
Fetch any site’s robots.txt, review its crawl rules and sitemaps, and test whether a specific URL path is allowed or blocked for search engines.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search? Allow: /admin/help/ Sitemap: https://example.com/sitemap.xml
What is robots.txt?
robots.txt is a plain-text file at the root of your domain that tells search engine crawlers which parts of your site they may or may not crawl. A misconfigured robots.txt is one of the most common — and most damaging — technical SEO mistakes: a single stray Disallow: / can deindex an entire website.
What to check
- Make sure you’re not accidentally blocking important pages or your whole site.
- Declare your XML sitemap with a
Sitemap:directive. - Remember: robots.txt controls crawling, not indexing — use a
noindexmeta tag to keep a page out of search results.
Use the path tester above to confirm a specific URL is crawlable before you publish.
Frequently asked questions
Is the robots.txt tester free to use?
Yes, it is completely free with no signup. Fetch a site's robots.txt and check whether a path is allowed or blocked for a given user-agent as many times as you need.
Why is a page still showing in Google if it's blocked in robots.txt?
Robots.txt only stops crawling, not indexing, so a blocked URL can still appear in results if Google finds links to it elsewhere. To keep a page out of the index, allow crawling and add a noindex meta tag or X-Robots-Tag header, or restrict it with authentication.
Where should the robots.txt file be located?
It must sit at the root of each host, such as https://example.com/robots.txt, and applies only to that exact domain, subdomain, and protocol. A file in a subfolder is ignored, and subdomains like blog.example.com need their own robots.txt.
Embed this free tool on your site
Add this robots.txt tester to your own blog or website — it stays free for your visitors, and the short credit line links back here. Just copy the snippet and paste it into your page’s HTML.
<iframe src="https://seoaiaudits.com/embed/robots-txt-tester" width="100%" height="700" style="border:1px solid #e5e7eb;border-radius:12px;max-width:760px;width:100%" loading="lazy" title="Robots.txt Tester"></iframe> <p style="font-size:13px;font-family:system-ui,sans-serif;margin-top:8px;color:#6b7280"> Free <a href="https://seoaiaudits.com/tools/robots-txt-tester" target="_blank" rel="noopener">Robots.txt Tester</a> by <a href="https://seoaiaudits.com" target="_blank" rel="noopener">SEO AI Audits</a> </p>
Tip: adjust height if your layout needs more or less room. Please keep the credit line so others can find the tool.
Want a different one? See every tool you can embed.
Related free tools
Instant SEO Snapshot
One-click SEO & AI-readiness grade for any page — HTTPS, indexability, title, meta, headings, content & more. Embeddable on your own site.
AI Bot Access Checker
What does your robots.txt say to OAI-SearchBot, PerplexityBot, Claude-SearchBot, GPTBot and the rest? Read with the real rules.
AI Model Recall Checker
Ask free AI models the unbranded questions your buyers type, and count how many name your brand unprompted — with each model and its training cutoff shown. It measures what those models remember from training, not a live check of ChatGPT, Claude or Perplexity; a built-in panel lets you spot-check the real engines by hand.
NLP Content Analyzer
Topics, entities, search intent & content gaps an NLP model sees in your page.
AI Retrieval Checker
AI-retrieval readiness for one page — an AI model’s read of how findable and quotable it is, with fixes.
AI Crawler Log Analyzer
Upload your access log to see which AI bots (GPTBot, ClaudeBot, PerplexityBot) actually crawled you — with IP verification where the operator publishes ranges. Runs in your browser.
Want the full picture?
This tool checks one thing. Run a complete, free SEO audit across 28 modules.
Run a free SEO audit