What is llms.txt? The format, who uses it and whether AI actually reads it

llms.txt is a proposed web standard: a Markdown file at yoursite.com/llms.txt that gives large language models a short summary of your site and a curated list of links to the pages worth reading, ideally in clean Markdown. Jeremy Howard of Answer.AI published the proposal at llmstxt.org on 3 September 2024. It is widely adopted on developer documentation sites, but it is not an official standard, and Google says Google Search, including AI Overviews, does not use it.

Quick answer

Why llms.txt exists

A large language model reading a web page has two problems. The page is full of navigation, scripts, cookie banners and ads that waste tokens, and a whole site is far bigger than any model's context window. The llms.txt proposal addresses both: it asks site owners to publish one small, human-written index that says what the site is and where the important material lives, so an agent can fetch only what it needs.

The proposal itself puts it simply: "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content," and notes that "the file itself stays small enough to fit in context." The point is use at inference time, when a person or an agent asks a question and a tool needs to read your docs now, rather than training data collection.

The llms.txt format, step by step

The file is plain Markdown in a fixed order. Per the spec at llmstxt.org:

1. An H1 with the name of the project or site. This is the only required part.
2. A blockquote with a short summary containing the key information needed to understand the rest of the file.
3. Optional detail sections (paragraphs or lists, but no headings) with anything else a model should know, such as conventions or caveats.
4. Zero or more H2 sections that are "file lists": each item is a Markdown link [name](url), optionally followed by a colon and a note about the page.
5. An optional H2 called Optional, which by convention holds secondary links an agent can skip when it is short on context.

A minimal example for a fictional product:

# Acme Queue

> Acme Queue is an open-source job queue for Python. This file lists the docs an AI assistant needs to answer questions and write working code against version 3.

Use the async API for new projects; the sync API is kept for compatibility.

## Docs

- [Quick start](https://acme.example/docs/quickstart.md): install, first worker, first job
- [Configuration](https://acme.example/docs/config.md): every setting with defaults
- [API reference](https://acme.example/docs/api.md): classes and methods

## Optional

- [Changelog](https://acme.example/changelog.md)

Markdown versions of pages

The proposal also recommends that pages offer a clean Markdown copy at the same URL with .md added (page.html.md, or index.html.md for a URL without a file name). The current version of the spec adds discovery hints: a page can point to its Markdown copy with rel="alternate" type="text/markdown" and to the llms.txt that covers it with rel="describedby", as a <link> element or an HTTP Link: header. It also allows llms.txt files at subpaths, covering the URLs below them, with agents using the most specific one.

What about llms-full.txt?

You will often see llms-full.txt next to llms.txt. It is not in the llmstxt.org proposal; it is a companion convention where a site concatenates all its documentation into one large Markdown file. It suits tools that want everything at once and models with big context windows. Anthropic's and Perplexity's documentation sites both serve one.

llms.txt vs robots.txt vs sitemap.xml

llms.txt does not block or allow anything. If you want to keep AI crawlers out, that is still robots.txt (and the specific user agents each vendor documents).

Who uses llms.txt

Adoption is strongest where the readers are developers and their coding agents. On 18 October 2026 we fetched the file from several sites and each one returned a real llms.txt: Anthropic's developer docs (platform.claude.com/llms.txt), OpenAI Developers (developers.openai.com/llms.txt), Perplexity's API docs, Stripe's docs, Vercel and the Model Context Protocol site. Stripe's file is a good model of the format's real value: as well as links, it carries instructions for agents, such as checking the package registry for the current version instead of trusting a version number remembered from training.

The tooling has followed. llmstxt.org lists documentation platforms and CMS plugins that generate the file (Mintlify, GitBook, Yoast SEO, AIOSEO and Wix among them), plus plugins for VitePress, Docusaurus and Drupal, and an MCP server that lets agents fetch and search llms.txt files. Community directories such as llmstxt.site and directory.llmstxt.cloud track sites that publish one.

Do search engines and AI models actually use llms.txt?

This is where most articles overstate things. Here is what is confirmed and what is not.

Confirmed: Google Search does not use it. Google's guide to optimizing for generative AI features, published in May 2026, lists llms.txt among things "you can ignore for Google Search." It says you don't need to create "new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." A note added in June 2026 says it is fine to keep one for other services, but doing so "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." That covers AI Overviews and AI Mode.

Confirmed: Chrome's Lighthouse checks for it. Lighthouse has an Agentic Browsing audit for llms.txt. Its documentation calls the file "an emerging convention" and says providing it is "optional at the moment": a missing file is marked not applicable, and the audit flags server errors when fetching it. This is a developer tooling check, not a ranking signal.

Confirmed: agents read it when pointed at it. Coding assistants and agent frameworks can fetch any URL you give them, and a llms.txt is a cheap, structured starting point. This is the use case the proposal was written for, and it works today with no vendor support needed.

Not confirmed: AI crawlers or answer engines using it as a signal. We found no published documentation from OpenAI, Anthropic, Google, Perplexity or Microsoft saying their crawlers or AI search products read llms.txt to decide what to cite or how to rank. Several of them publish llms.txt for their own docs, which shows they find it useful for developers, not that their search systems consume it. Treat any claim that llms.txt "gets you cited in ChatGPT" as unproven.

Should you add one? A practical checklist

Add one if you have documentation, an API, an SDK or a product that people build against. It is the cheapest way to make your docs agent-friendly, and the same discipline helps with context engineering in your own agents.

1. Write the H1 and a two to three sentence summary a model could quote.
2. List 5-30 of your most useful pages, each with a one-line note. Curate; don't dump your sitemap.
3. Serve Markdown copies of those pages if your platform can (many doc platforms do it automatically).
4. Put rarely needed material under ## Optional.
5. Return it as text/plain or text/markdown with a 200 status, and keep it current when docs move.
6. Test it: paste the URL into an AI coding assistant and ask a question only your docs answer.

Skip the effort, or keep it minimal, if your goal is Google rankings or AI Overview citations. For those, Google's advice is ordinary SEO: crawlable pages, clear structure and original, useful content.