LLMention
Setup

How to Generate and Deploy an llms.txt File

A practical guide to structuring, hosting and verifying an /llms.txt file at your domain root, and an honest look at what it does not do for AI visibility.

By Alex Xu3 min read

llms.txt is a plain markdown file served at your domain root that tells language models which parts of your site matter. It is a convention, not a standard, and it takes about ten minutes to add. This guide covers the format, the deployment, and how to verify it is actually reachable.

What robots.txt, sitemap.xml and llms.txt each decideA crawler arrives and meets three files in order. robots.txt decides whether it may fetch at all. sitemap.xml tells it which pages exist. llms.txt tells it what the site is about once it is reading. Each answers a different question and none replaces another.A crawler arrivesrobots.txtsitemap.xmlllms.txtMay it fetch?Permission.Empty file = allowed.What exists?Discovery.Missing = still crawled.What is it?Comprehension.Missing = inferred.None of the three substitutes for another.An llms.txt listing pages does not make them discoverable, because discovery is sitemap.xml.A sitemap submitted does not grant permission, because permission is robots.txt.
Read left to right in the order a crawler meets them: permission, then discovery, then comprehension.

What is llms.txt?

/llms.txt is a proposed convention for giving AI systems a curated, machine-readable summary of a website. Instead of asking a model to infer structure from HTML, it hands over a short markdown document with a project summary and a list of the most important links. The companion file /llms-full.txt carries the expanded content itself.

What goes inside the file?

The convention is deliberately simple: one H1 with your site name, a blockquote summary, then H2 sections of annotated links.

# Your Company

> One or two sentences describing what you do and who it is for.

## Core pages
- [Product overview](https://example.com/product): what it does.
- [Pricing](https://example.com/pricing): plans and limits.

## Documentation
- [Getting started](https://example.com/docs/start): setup in five minutes.

## Optional
- [Blog](https://example.com/blog): release notes and research.

Two rules matter more than the format: the summary line is what a model is most likely to quote, so write it as a fact-rich sentence rather than a slogan; and every link must resolve, because a file full of 404s is worse than no file at all.

Where does the file go?

It must be reachable at the exact path https://your-domain.com/llms.txt — the domain root, not a subdirectory. How you publish it depends on your stack: on Next.js, drop it in public/llms.txt; on most static hosts, put it in the publish directory. Avoid redirects: serve it directly with a text/plain content type.

Does llms.txt actually improve AI visibility?

We can show what it costs; we cannot show that it improves visibility. Neither of the crawler documents we could open — OpenAI’s help centre and Anthropic’s crawler article, dated 7 April 2026 — names the file; both specify robots.txt as the mechanism they honour.

What it genuinely helps with today: documentation sites and coding agents that fetch context deliberately, and any retrieval system that accepts a curated entry point. The honest framing is one small, cheap signal — worth adding, not worth treating as a ranking factor.

How do I verify it is live?

  1. Request https://your-domain.com/llms.txt in a browser. It should render as plain text, not download or 404.
  2. Confirm the response is 200 and the content type is text/plain.
  3. Check that robots.txt does not disallow the path, and that your firewall does not block crawler user-agents.
  4. Click every link in the file once. Broken links are the most common defect.

You can run all of this automatically with the llms.txt Studio, or audit crawler access first with the free GEO scanner.

Questions about deploying llms.txt

Where exactly does the file go?

At the domain root, so that https://your-domain.com/llms.txt returns it directly with HTTP 200 and a body. A redirect to a login page, or an HTML error page, counts as not served.

Does publishing llms.txt improve rankings?

We have not found a crawler document that commits to reading it; the two we could open, OpenAI's help centre and Anthropic's crawler article, specify robots.txt instead. It is cheap and worth doing for the readers that do fetch it, and we do not treat it as a ranking factor.

How do I confirm it is being served correctly?

Request the URL and read what comes back: markdown with a 200, rather than a 404 or a page of HTML. A status code alone is not enough, because an empty file returns 200 too.

Evidence and sources

GEO can boost visibility by up to 40% overall in generative engine responses, and in the paper's Table 1 the best of the tested methods improve on the no-optimization baseline by 41% and 28%. — Generative Engine Optimization, KDD 2024

The weighting this site uses follows that measurement, which is why citability and evidence carry 11% while the AI context file dimension carries 5%. The full weighting, and the parts of the picture a single-URL scan cannot see, are on the methodology page.

Primary sources