What is llms.txt?

The proposal defines a Markdown file named llms.txt at a site’s root (/llms.txt) or under a subpath such as /docs/llms.txt. Each file applies to the pages beneath its location, and when more than one applies, agents should prefer the most specific^[1]. Its parts come in a fixed order:

• an H1 giving the site or project name, the only required part;
• a blockquote with a short summary;
• optional paragraphs or lists, but no headings;
• optional H2 sections, each a list of links with an optional note per link.

By convention, an H2 section titled “Optional” holds secondary links an agent may skip when it needs a shorter context^[1].

The proposal also asks sites to publish clean Markdown copies of their pages at the same URL with .md appended (page.html.md) or with the extension replaced by .md (page.md). Version 2 adds two link relations to help agents find them: rel="alternate" with type="text/markdown" links a page to its Markdown copy, and rel="describedby" links it to the llms.txt file that covers it^[1][2]. Agents are expected to read or search the file first, then follow its links^[1].

• Google Search ignores it. Google says creating llms.txt files “will neither harm nor help” a site’s visibility or rankings in Google Search, including its generative AI features^[3].
• Chrome’s Lighthouse checks for it. Lighthouse has an llms.txt audit that flags a server error when the file is requested and marks a missing file “Not Applicable,” because providing it is “optional at the moment”^[4].
• AI companies publish it but don’t say they read it. OpenAI, Anthropic and Perplexity each serve an llms.txt file for their own developer documentation^[9][10][11]. Their crawler documentation, and Apple’s, does not say their crawlers read llms.txt files on other sites (checked September 23, 2026)^[5][6][7][8].

The proposal’s author writes that thousands of sites now publish the file and that coding agents “use them reliably”^[2]. Those are his statements, not independent measurements.

Request /llms.txt on the site in a browser and confirm that it returns the Markdown file rather than an error page. For an automated check, Chrome’s Lighthouse llms.txt audit flags a page when the server returns an error for the file and treats a missing file as not applicable^[4].

• Myth: llms.txt helps a site rank in Google or appear in AI Overviews. Google says Google Search, including its generative AI features, doesn’t use the file, so it neither helps nor harms visibility there^[3].
• Myth: llms.txt can block AI crawling or training. The proposal says robots.txt and llms.txt do different jobs: robots.txt states what automated access is acceptable, while llms.txt offers information when an agent needs it^[1]. OpenAI’s documented way to keep content out of model training is a robots.txt rule for GPTBot^[5].
• Myth: sites are required to have one. Lighthouse marks a missing llms.txt as not applicable because the file is optional for now^[4], and Google Search ignores it^[3].

Jeremy Howard published the proposal in September 2024; the page carries the date 2024-09-03^[1][2]. In August 2026 he revised it as version 2, adding the two discovery link relations, allowing page.md as well as page.html.md, defining how files in subpaths apply, and dropping the special machine meaning of the “Optional” section^[2].