What Is llms.txt? A Practical Guide for 2026

I want to be upfront about something before we go any further. llms.txt is not a confirmed ranking signal, and the evidence that anything reads it is thin. A 2026 Ahrefs server-log study found 97% of llms.txt files were never requested at all (2026-06-15, read 2026-09-23), and Google's AI optimization guide says you don't need to create AI text files or markup to appear in Google Search, including its generative AI features (last updated 2026-07-10, read 2026-09-23). No major AI search engine has committed to reading it the way they honour robots.txt, and products that fetch pages live may ignore it entirely.

With that said: it takes about 20 minutes to make a good one, the cost of getting it wrong is zero, and it is genuinely useful for AI coding agents and documentation sites: the tools a developer points straight at your docs. That is the case for having one, and it is enough.

Here's what it actually is and how it works.

What problem is it solving?

When an AI assistant tries to understand a website, it runs into the same walls a human skimmer does, only worse. A typical HTML page is buried in navigation menus, cookie banners, sidebar widgets, and JavaScript that the model may never execute. Pulling a useful answer out of that takes token budget that could have gone toward understanding your actual content.

llms.txt is a workaround. Instead of leaving the model to guess which of your 500 URLs matter, you write one short file that says in plain prose: here is who we are and here are the twelve pages worth reading. It's a table of contents written for language models instead of crawlers.

It's also worth being clear about what it is not. robots.txt tells crawlers what they can't access, and an AI crawler you've blocked there won't read your llms.txt either. sitemap.xml lists every URL for completeness. llms.txt is the opposite of completeness: it's a deliberate shortlist, written in prose a model can read directly.

What the format looks like

The spec is intentionally minimal: standard Markdown with a loose structure. Any model that can read text can parse it.

Each link is a standard Markdown list item: - [Page name](url): one-line description. The description after the colon is the part that makes the file useful. It tells the model what it'll find on the page before spending a request to fetch it.

A minimal but complete example:

# Acme Analytics

> Acme is a privacy-first web analytics tool for small teams.

## Docs

- [Quickstart](https://acme.com/docs/quickstart): Install and see your first dashboard in five minutes.
- [API reference](https://acme.com/docs/api): Full REST endpoints with authentication and rate limits.

## Optional

- [Changelog](https://acme.com/changelog): Release notes, useful but not essential reading.

There's also a convention for an llms-full.txt file that inlines the full content of those pages into one document, so a model can read everything in a single fetch. That file grows large quickly. Most sites are better off with the lean llms.txt and letting models follow links as needed.

Why it actually helps

Three concrete reasons, none of them magic:

It's more efficient. Models work inside a fixed token budget. Pointing straight at clean, relevant pages means less budget spent parsing navigation and boilerplate, and more on your actual content. On the pages themselves, semantic landmarks like `<main>` and `<nav>` do the same job of separating your content from the furniture around it.

It improves how you're framed. When you write the one-line descriptions yourself, you control how your pages are introduced to the model. It's less likely to misrepresent what you offer if you've told it plainly what each page covers.

It is cheap, and it is read where it counts. AI coding agents and documentation tools that a developer points at your site will read a curated Markdown index and use it. That is the audience to write it for. What publishing the file will not do is get you cited: no AI search engine documents reading it, and being quoted in an AI answer is on-page work, which I cover in how to get cited by ChatGPT, Perplexity and AI Overviews.

How to make one

You can write it by hand in about 20 minutes:

1. Pick your highest-value pages. Documentation, pricing, key product pages, your best guides. Aim for 10 to 30 links, not your entire sitemap.
2. Write the title and summary. One H1 with your name, one blockquote that explains what you do in a sentence.
3. Group the links under H2 headings. Cluster by purpose: Docs, Guides, Products, About. Write a short, factual description for each link.
4. Move nice-to-have links under Optional. This tells a model what it can safely skip if it's running low on budget.
5. Put it at /llms.txt on your domain and serve it as text/plain.

If you'd rather start from a generated draft than a blank file, the llms.txt generator builds a spec-compliant version from your URL, with first-pass descriptions you can edit.

How to validate it

A file that exists but is malformed doesn't help anyone. Two things to check. Structure: an H1 title, at least one link section, valid Markdown link syntax, and URLs that actually resolve. Delivery: the file must be reachable at /llms.txt and served as plain text, not wrapped in your site's HTML.

The fastest check is the llms.txt validator, which fetches your file, checks it against the spec, and flags broken links, missing sections, or wrong content types. Worth re-running whenever you restructure your site, because a stale file pointing at links that have since broken is worse than none.

How many sites actually have one?

Our own audit data gives a rough answer. Of the 123 websites audited on SEO AI Audits between 25 August and 8 October 2026, 58 (47%) served a real llms.txt file when we fetched each one on 8 October. That is far more than we expected, and it says more about tooling than about demand: SEO plugins now offer to generate the file with a single setting.

The caveat that keeps the number honest: these are sites people chose to audit with a free SEO tool, so they lean towards small sites whose owners already care about SEO, and 123 sites is a small sample.

What's inside the files

On 10 October we downloaded the files again (59 were up) and read them. A third of them weren't really separate files: 20 were copies of three templates, repeated across small AI-tool sites. Leave those out and adoption is still about 37%. The figures below cover the 39 files that aren't copies.

• The format is mostly right. 34 open with the # Site name heading the spec asks for, 38 group their links under ## sections, and 29 have the > summary line. Only 10 use an ## Optional section, and just 1 also publishes an llms-full.txt.
• The descriptions are the weak spot. Of the 37 files with link lists, 17 describe every link, 11 describe none, and 9 describe some. A bare list of titles and URLs tells a model nothing a sitemap doesn't.
• Plugins wrote 11 of the 39: Yoast 6, All in One SEO 3 and Rank Math 2. Yoast's were small and tidy, but five of the six described no links. Rank Math's described nearly every link. All in One SEO's ranged from a 37-link file to two that listed more than a thousand pages each.
• Some links are dead. We checked up to 20 same-site links per file: 4 files pointed at pages that now return 404, and in two of them most of the checked links were dead.
• Nobody links to Markdown. The proposal suggests offering a clean .md copy of each useful page; none of the 59 files linked to one.

The median file had 19 links and weighed under 5 KB, which is about right. If your file came from a plugin, read it once and add a plain sentence to each important link, because that is what turns a link list into something a model can use. The full method, with a one-line command to check your own file for dead links, is in our write-up on dev.to.

We publish the wider picture, the most common issues across every site we audit, on our research page.

Should every site have one?

Documentation sites and developer tools get the clearest benefit, and for everyone else the cost is close to zero, so publish one if you would like the tools that do fetch it to read you accurately. Just not as a route to AI search traffic: that comes from the content on your pages — answering the question directly, in the first sentence under the heading.

Keep it lean and keep the descriptions honest. A short, accurate file beats a sprawling one that no longer matches your site. Revisit it when things change.

Want to see how AI-ready your whole site is beyond this one file? A free SEO audit checks your structure, machine-readable signals, and content clarity in about a minute, no signup required.