Free Robots.txt Generator

Create a valid robots.txt file in seconds. Choose your crawl policy, add disallow paths, set a crawl-delay, and declare your sitemap — then copy and upload to your site root.

robots.txt
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?

Sitemap: https://example.com/sitemap.xml

Save as robots.txt in your site root. Remember: this controls crawling, not indexing.

Example resultA sample robots.txt for example.com — generate one for your own site above for a copy-ready file.
Crawl policy
Allow crawling
Disallow paths
3 rules
Crawl-delay
10s
Sitemap
Declared
User-agent
* (all)
Validity
Valid
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Robots.txt best practices

  • Always include a Sitemap: line pointing to your XML sitemap.
  • Be careful with Disallow: / — it blocks your entire site from search engines.
  • Use it to keep crawlers out of admin areas, carts, and parameter URLs — not to hide pages (use noindex for that).
  • Place the file at your root: https://yoursite.com/robots.txt.

Already have one? Test it with our free robots.txt tester.

Frequently asked questions

Where do I put the robots.txt file on my site?

It must go in the root directory of your domain so it is reachable at yourdomain.com/robots.txt. Crawlers only check that exact location, so it will not work in a subfolder, and each subdomain needs its own robots.txt.

Does Disallow in robots.txt stop a page from showing up in Google?

No. Disallow only asks crawlers not to crawl a page, but a blocked URL can still be indexed if other sites link to it, often shown without a description. To keep a page out of results, allow crawling and use a noindex meta tag or HTTP header instead.

Should I add my sitemap URL to robots.txt?

Yes. A Sitemap line with the full URL of your XML sitemap helps search engines discover your pages, and this generator includes it for you. It is independent of allow and disallow rules, so you can list one or more sitemaps either way.

Related free tools

Keyword & Question Idea Generator

Hundreds of real questions, comparisons & related searches people type — from Google, Bing & DuckDuckGo autocomplete. AnswerThePublic-style, free.

Schema Markup Generator

Generate valid JSON-LD structured data — Article, Product, Org, LocalBusiness, Breadcrumb, HowTo & more.

FAQ Schema Generator

Turn questions & answers into valid FAQPage JSON-LD — structured answers ChatGPT, Perplexity, AI Overviews & Bing can cite.

llms.txt Generator

Generate a spec-compliant /llms.txt — handy for AI coding tools & agents (an emerging convention, not a proven search-citation lever).

Meta Tag Generator

Generate title, description, Open Graph & Twitter Card tags to copy.

SERP Snippet Preview

See your Google result on desktop & mobile with pixel-width truncation.

Want the full picture?

This tool checks one thing. Run a complete, free SEO audit across 26 factors in about a minute.

Run a free SEO audit