llms.txt is the rare thing in this field: a convention with almost no standardisation, a very low cost of adoption, and a lot of claims attached to it. It is also the file most likely to be published for the wrong reason — because a checklist said to, not because anyone has a use for it. This article is about what the file can do, what it cannot, and how to tell whether yours is worth the bytes.
The short version: publish it if you want to, keep it accurate, and do not expect it to change your access. The work that changes access is the crawl path and the content itself.
What the file is
The convention proposes a Markdown file at /llms.txt that gives a model a curated map of a site: a title, a one-line summary, and sections of links with short descriptions. An optional /llms-full.txt carries a fuller corpus for tools that want the text rather than the map. The details come from the proposal at llmstxt.org, and the scanner tests only the shape, not the prose.
That is the whole mechanism. There is no registry, no crawler instruction that makes a model read it, and no ranking system that rewards it. An agent may fetch the URL because its instructions say to look for one; if it does, a good file saves it work. If it does not, the file is still a URL on your site that costs almost nothing.
What the scanner checks — and the number it is worth
The rule is published in full: the file must return more than 20 bytes, contain an H1, contain a > summary line and contain at least three Markdown links. Missing any of those is a partial pass. What the file says is not judged, which the rule page states plainly: three links to pages that do not exist pass the structure test.
The points are deliberately small. The rule is worth 4 of the 6 points in the AI context files dimension, and the dimension is 5% of the total — so the file can move a score by at most about 3.3 points out of 100. The methodology page says the reason: the convention is cheap to follow, and the evidence that engines consume it is weak. A site that treats this file as the main lever is optimising the cheapest thing on the list.
What it changes
- If an agent reads it: it gets a curated list of what matters, in the order you chose, with one-line descriptions. That is a better starting point than guessing from a sitemap, which lists URLs without saying which ones are the product and which are the legal pages.
- If a human reads it: the file is a compact statement of what the site is. That has some value on its own — it is a page that can be linked, quoted and diffed.
- If nobody reads it: nothing happens. That is the honest default, and it is why the cost matters more than the upside.
What it does not do
The file is not a permission system. It does not allow a crawler that robots.txt or a WAF refuses, it does not render JavaScript, and it does not create content that is not already published. It also does not express training consent in a way that any major provider has agreed to honour; that discussion happens in robots.txt, terms, and the crawler's own documentation, with varying degrees of enforceability.
Nor is it a substitute for the files beside it. Sitemaps describe URLs, robots.txt describes crawl permission, structured data describes entities and page types. A context file describes the site to a reader that has already decided to look. Publishing one does not remove the need for the other three, and deleting the others while adding this one is a net loss.
A file that is worth the bytes
The shape is simple. Keep it Markdown, keep the summary one sentence, and write each section for a reader who has never seen the site.
# Your product name > One sentence on what this site is and who it is for. ## Core pages - [Product overview](https://your-domain.com/): what it does, in one line. - [Pricing](https://your-domain.com/pricing/): what is free and what is paid. - [How it works](https://your-domain.com/methodology/): the method and its limits. ## Reference - [Rules](https://your-domain.com/checks/): every rule the tool applies. ## Optional - [Changelog](https://your-domain.com/changelog/): what changed, dated.
Three properties do most of the work: absolute URLs, descriptions that say something the link text does not, and a stable order that puts the most useful pages first. The optional section is the right place for pages that are useful but not central; it is not a place to hide the product.
Common mistakes, in the order they cost you
- Serving HTML at
/llms.txt. A framework catch-all route returns the app shell with a 200, the file fails every structural check, and the reader gets a page where a file was promised. Check the content type and the first bytes. - Listing pages that do not exist. The scanner cannot catch this because it does not follow the links; a reader can, and a broken map is worse than no map.
- Writing marketing instead of a map. The summary is one sentence; the sections are links. A page of adjectives tells an agent nothing it can use.
- Letting it drift. A dated file that still lists the old pricing page is a false statement about the site, and it is the most common failure of hand-maintained files. Either generate the list from a source of truth or put the review date in the CI checklist.
- Expecting it to fix access. If a crawler is refused a page, the context file cannot serve the content it is refused. Fix the refusal first; the article on 403s covers how to tell which layer is refusing.
How to test yours in four commands
# 1. Status and content type curl -sS -D - -o /dev/null https://your-domain.com/llms.txt # 2. It should start with a heading, not HTML curl -sS https://your-domain.com/llms.txt | head -n 3 # 3. Count the markdown links curl -sS https://your-domain.com/llms.txt | grep -cE '\[[^]]+\]\(https?://' # 4. Find links worth checking by hand curl -sS https://your-domain.com/llms.txt | grep -oE 'https?://[^)]+' | sort -u
Command 3 is a proxy for the scanner's structure test; command 4 gives you the list to check against a sitemap or a link checker. A file with a heading, a summary and three links passes the rule. A file with those and no broken links is actually useful.
What to expect from it
Nothing measurable, in the short term. If your server logs show requests for /llms.txt, something is reading it; if they do not, nothing is. The file is best treated as a low-cost convention with a diagnostic side effect: it forces a site to state, in one page, what it is and which URLs matter.
The higher-value work is unchanged: make the pages fetchable, put the content in the response, and structure it so an answer can be extracted. A context file is a nice description of a site that already works. It is not a substitute for one.