Creating and Using llms.txt

llms.txt in one sentence

The llms.txt is a simple Markdown file in the root directory of your website (your-domain.com/llms.txt), which gives large language models (LLMs) a curated, easily readable overview of your site’s most important content. It is meant to help AI systems find and correctly understand your most relevant pages quickly – without having to dig through cluttered HTML code, navigation, and ads.

The llms.txt was proposed in September 2024 by Jeremy Howard, the co-founder of Answer.AI. Important to note right away: it is a proposed community standard, not a specification officially confirmed by Google, OpenAI, or Anthropic. We clarify honestly further down what that means in practice.

Why the llms.txt exists

LLMs work with a limited context window – they cannot process an unlimited amount of text at once. A normal HTML page contains a lot of “noise”: navigation, footer, scripts, ads, cookie banners. For a model, filtering out the actual substance from that is laborious.

The llms.txt solves this by providing the core of your website as a clean Markdown list: the most important pages, each with a title, link, and short description. Markdown is deliberately chosen because it is easily readable for both humans and machines and conveys structure without ballast.

llms.txt vs. robots.txt vs. sitemap.xml

The three files are often confused, but they serve completely different purposes:

In short: the robots.txt says “entry allowed or forbidden,” the sitemap.xml says “here are all my URLs,” and the llms.txt says “this is what matters most, and here is how it connects.” The llms.txt replaces neither robots.txt nor sitemap.xml – it complements them. How to control AI crawlers via the robots.txt is covered in the article Controlling GPTBot and AI bots via robots.txt; the basics of the file are explained in the reference article robots.txt.

How an llms.txt is structured

The specification is deliberately lean. An llms.txt follows this structure:

• An H1 heading with the name of the website or project (the only mandatory element).
• An optional blockquote (>) with a short summary.
• Any number of detail paragraphs for context.
• Several H2 sections with thematically grouped link lists. Each link consists of a Markdown link plus an optional description after a colon.
• Optionally a section ## Optional for content an AI may leave out when space is tight.

In addition, there is the llms-full.txt variant: it contains not just the links, but the full content of the most important pages as Markdown, so a model can load everything in one go. For large websites, this full version quickly becomes unwieldy – here the lean, linked llms.txt is usually the better choice.

Another frequently used pattern: many providers make a Markdown variant available in addition to each important HTML page (reachable at the same URL with the extension .md). This lets an AI load the clean text version instead of the HTML page. This is not part of the llms.txt specification itself, but in practice it is often used alongside it.

Practical template

This is what an llms.txt for a German-language agency website could look like:

How to create and add the llms.txt

1. Select core content: determine the 10 to 30 most important pages an AI should know about you – services, key guides, about-us.
2. Write Markdown: set up the structure as in the template, with descriptive titles and concise descriptions.
3. Place the file: save it as llms.txt in the root directory, so that it is reachable at your-domain.com/llms.txt.
4. Keep it current: update the file when important pages are added or removed. For larger websites, a generator that produces the file automatically from the CMS is worthwhile.
5. Do not mistake it for access control: the llms.txt neither allows nor blocks crawlers. Anyone who wants to protect content still uses robots.txt and server-side rules for that.

An honest assessment: adoption in 2026

Here is the most important point for a realistic expectation. As of 2026, no major AI provider has officially confirmed that its crawlers or retrieval systems actively read the llms.txt and prioritize it. Google expressed public skepticism and compared the idea to the long-meaningless keywords meta tag. So far, adoption is strongly community- and developer-driven – above all providers of developer documentation and technical tools make an llms.txt available.

What does that mean for you? The llms.txt is not a guaranteed ranking or citation lever. It is a low-risk, quickly implemented measure with uncertain but potentially growing value: the effort is small, it does no harm, and should the standard catch on, you are in early. Anyone who wants to build thought leadership today can make the file available – but without treating it as a substitute for the substantial GEO levers such as citable content, structure, and authority.

Conclusion

The llms.txt is a lean Markdown file that gives AI systems a curated overview of your most important content. It differs clearly from robots.txt (access) and sitemap.xml (URL list) and is quick to create. As of 2026, however, its real value is still open, because no major AI provider has officially confirmed evaluating it. As a low-risk complement with a future option it is worthwhile – as the foundation of your GEO strategy it does not qualify. Set up the file, but invest your main budget in content and authority.