llms.txt: what belongs in it and what it actually does
llms.txt is a plain markdown file at your site root that gives an AI agent a short, curated map of what matters on the site. It is a proposal from 3 September 2024, not a standard, and no assistant is obliged to read it. It costs an hour to write and it cannot hurt, which is the honest case for it.
What the file actually contains
The specification is short. A file begins with an H1 heading, which is the only element the spec calls truly mandatory. Below it goes a blockquote with a brief summary of the project, then optional paragraphs of detail, then zero or more H2 sections each holding a list of links.
Each list entry follows one pattern: a required markdown hyperlink in the form [name](url), then optionally a colon and a note about the file. That is the whole grammar. The file may sit at the site root as /llms.txt or at any path, covering the pages beneath that path.
Why markdown and not XML
The proposal is explicit that it uses markdown to structure the information rather than a classic structured format such as XML, which is unusual for a machine readable file. The reason is the reader. A sitemap is parsed by a crawler, an llms.txt is read by a language model, and a language model reads markdown natively while XML costs it tokens for no gain.
The stated purpose is to offer concise, expert level information gathered in one accessible location, so an agent does not have to spend its context window crawling a navigation menu to find out what a site is for. That framing tells you what belongs in the file: the handful of pages you would show a competent stranger, not an export of every URL you own.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Format | Plain text directives | XML | Markdown |
| Read by | Crawlers | Crawlers | AI agents, voluntarily |
| Says | What may be fetched | What exists | What matters and why |
| Enforced | Widely honoured | Widely honoured | By nobody |
| Age | Since 1994 | Since 2005 | Proposed 2024 |
What we found across 187 live hosts
We audited our own network in July 2026, 187 live hosts in total, and the result was lopsided. llms.txt existed on 6 of them. Not a single host blocked an AI crawler, and 25 of the 187 served no robots.txt at all, which is a more consequential gap than a missing llms.txt.
Two conclusions follow. The first is that llms.txt is still rare enough that writing one costs nothing competitively; you are not catching up with anybody. The second is about order of work. A missing robots.txt affects every crawler on the open web, a missing llms.txt affects a voluntary convention proposed in 2024. If you have an hour, spend it on the file that has been honoured since 1994.
The honest limits
Nobody is obliged to read it. There is no search engine that promises to honour it, no penalty for omitting it and no measurable ranking effect anyone has published. Any agency selling llms.txt as an AI ranking factor is selling something that does not exist.
What it does do is cheap and real. It gives one clean, curated statement of what your site is and which pages matter, in a place an agent can find without guessing. For a nine page business site that is twenty lines of text. Write it once, keep it honest, and treat it as insurance rather than as a lever.
Questions and answers
Does llms.txt improve my ranking?
Where exactly does the file go?
What is the minimum viable file?
Should I list every page?
Sources
- The llms.txt proposal, llmstxt.org Specification, structure and stated purpose; version 2 dated 10 August 2026
- Google Search Central, Google crawlers and user agents For contrast, on what robots.txt directives actually control
