llms.txt is a proposed convention for websites to publish a plain-text or Markdown file at the path /llms.txt, listing curated links and summaries intended to help AI systems and large language models find and understand a site's content more efficiently than by crawling and parsing its full HTML. It was proposed in September 2024 by Jeremy Howard, co-founder of fast.ai, explicitly modeled on the long-established robots.txt and sitemap.xml conventions that already give automated crawlers structured guidance about a site.
The proposal responded to a practical problem: modern web pages are often heavy with navigation, JavaScript rendering, advertising, and layout markup that make them costly or unreliable for a language model to process within a limited Context window, whereas a curated Markdown summary with direct links to key pages could be consumed cheaply and reliably.
Format and adoption
A typical llms.txt file opens with an H1 title and short summary of the site, followed by Markdown-formatted link lists grouped under headings, often pointing to a corresponding llms-full.txt file with complete page content rendered in Markdown. Adoption spread first among developer-tools companies and documentation sites, where clean, crawlable technical content was already a priority, before extending to broader consumer and content platforms through 2025 and 2026. Unlike robots.txt, which is enforced by convention across virtually all major crawlers, llms.txt has no equivalent universal support; as of the mid-2020s no major AI lab had confirmed that its models or crawlers systematically fetch and use the file, leading some commentators to describe adoption as largely speculative or symbolic.
Relationship to GEO and criticism
The proposal is frequently discussed alongside generative engine optimization, the practice of making content more likely to be cited by AI answer engines, since a well-maintained llms.txt is one candidate lever for GEO, even though its actual effect on citation rates is unverified. Critics have argued that in the absence of confirmed consumption by major AI systems, llms.txt functions more as a statement of intent or a hedge than a proven optimization technique, and that AI agents increasingly reach for full-page browsing and tool use rather than relying on a static curated summary. Supporters counter that even partial adoption reduces friction for the agents and retrieval pipelines that do respect it, and that the convention costs little to publish regardless of uncertain uptake.