llms.txt

Long definition

llms.txt aims to do for LLMs what robots.txt does for crawlers and sitemap.xml does for search engines: provide a structured, machine-readable hint about what's on a site and what matters most. The format is plain Markdown, served at https://example.com/llms.txt, and follows a loose template:

# Project Name

> One-paragraph summary of what this site is about.

## Docs

- [Getting started](https://example.com/docs/start.md): brief description.
- [API reference](https://example.com/docs/api.md): brief description.

## Optional

- [Changelog](https://example.com/changelog.md): brief description.

The "Optional" section is a convention — items there are deprioritized for context-limited LLM consumption. Authors are also encouraged to publish parallel .md versions of key pages so models can ingest clean Markdown instead of HTML-stripped versions.

Adoption ramped through 2025. Major early adopters include Anthropic (anthropic.com/llms.txt), Stripe, Cloudflare, Vercel, and most documentation tooling vendors. Adoption among general SEO sites is still low — most marketing sites have not added it as of early 2026, and major LLMs do not yet uniformly fetch the file at query time.

The unsettled part: llms.txt is not a standard recognized by Google, OpenAI, or Anthropic in the way robots.txt is. None of the major model providers commit to reading the file. Adoption today is closer to a structured-data signaling convention — you're betting models or downstream tools will consume it, the way schema.org bet on ranking benefits before Google adopted it broadly.

Cost-benefit: the file is cheap to produce and maintain. For documentation sites, technical products, or content libraries, the upside-to-effort ratio is favorable. For brochure marketing sites, the file does little until major LLMs commit to reading it.

Common misconceptions

• "llms.txt is an official W3C or IETF standard." It's not. It's a community proposal driven by Jeremy Howard and tracked at llmstxt.org. There's no governing body, no RFC, no formal commitment from major model providers.
• "Adding llms.txt makes ChatGPT cite me." It might help, eventually, if model providers adopt the file. Today, no major LLM commits to reading llms.txt at query or training time. Citation lift is unproven.
• "llms.txt replaces robots.txt or sitemap.xml." Different files, different purposes. robots.txt controls access. sitemap.xml lists URLs for discovery. llms.txt prioritizes content for LLM consumption. All three can coexist.
• "You need to maintain a parallel .md tree of your whole site." No. The recommended pattern is parallel .md versions of your most important pages — not every URL. Start with docs hubs, key landing pages, and reference content.