Industry Analysis · 5 min read
What is llms.txt, and is it worth adding to your site?
Short answer
llms.txt is a proposed Markdown file at your site root that gives language models a short summary and a curated list of your most useful pages. It is cheap to publish, but as of November 2026 Google says Google Search ignores it, so treat it as a convenience for AI tools and agents, not a ranking lever.
Key takeaways
- llms.txt was proposed by Jeremy Howard of Answer.AI in September 2024; the only required element is an H1 with your site or project name.
- It does not block or allow crawlers. Access control still lives in robots.txt and your server rules.
- Google's own guidance (June 2026) says Google Search ignores llms.txt, so publish it for developer tools and agents, and keep expectations modest.
Someone on your team has read that every site now needs an llms.txt file, and you want to know whether it is a real requirement, a nice-to-have, or noise. This page explains what the proposal says, what the file can and cannot do, what Google has said about it, and how to write a decent one without spending a week on it.
Where did llms.txt come from?
llms.txt is a proposal published by Jeremy Howard of Answer.AI on September 3, 2024, and documented at llmstxt.org.
The problem is practical. Language models have limited context windows, and a typical page wraps its useful text in navigation, scripts and banners. Howard suggested a short Markdown file at /llms.txt that summarizes the site and points to clean versions of key pages, plus Markdown copies of pages at the same URL with .md appended.
Answer.AI and fast.ai used it on their own documentation first. The format was designed for software docs that developers paste into coding assistants, and that is still where it earns its keep.
What goes in the file?
The file is plain Markdown with one required element, an H1 title, followed by an optional summary and sections of annotated links.
The spec at llmstxt.org lays out the order:
- An H1 with the name of the site or project (the only required part).
- A blockquote with a short summary that gives the key context.
- Optional paragraphs or lists with more detail, but no headings.
- Zero or more H2 sections, each holding a list of links in the form
[name](url): notes. - An optional H2 called "Optional", for secondary links an agent can skip when it needs a shorter context.
What about llms-full.txt?
You will see llms-full.txt on many documentation sites. It is a convention that grew up around the proposal: one file containing the full text of the linked pages. Howard's original post described similar generated files called llms-ctx.txt and llms-ctx-full.txt. For a services business with 30 pages, a full-text file adds little; for an API reference with 400 pages, it can save developers real time.
What is llms.txt not?
llms.txt is not an access control, not a replacement for robots.txt and, as of November 2026, not a confirmed ranking or citation signal for any major AI search engine.
Confusion usually comes from the name. Here is how it compares with the files people mix it up with:
| File | What it does | Enforced by |
|---|---|---|
| robots.txt | Tells crawlers which paths they may fetch | Voluntary compliance by each crawler |
| sitemap.xml | Lists URLs you want discovered and recrawled | Used as a hint by search engines |
| llms.txt | Summarizes the site and lists recommended pages for LLMs | Nothing; reading it is optional |
If you want to keep a bot out, llms.txt does nothing. Use robots.txt rules for the bot's user agent, or server-level blocking. Our GEO readiness checker shows which AI crawlers your robots.txt currently allows.
Google has been explicit. Its guide to generative AI features says you do not need machine-readable files, AI text files or Markdown to appear in Google Search, and that creating them will "neither harm nor help" visibility because Google Search ignores them. Google added that note to its documentation on June 15, 2026. We have found no documentation from OpenAI, Perplexity or Microsoft saying their search products use llms.txt for ranking or citations. Some agents and coding tools may fetch it, and that behavior varies by product.
Is it worth the effort?
For most business sites it is a one-hour task with a small, uncertain upside, so do it after the basics, never instead of them.
The honest cost-benefit looks like this:
- Cost: an hour or two to write, plus a quarterly review so links do not rot.
- Benefit: developer tools and agents that look for the file get a clean map of your site.
- Benefit: writing it forces you to name your ten most important pages, which is a useful exercise on its own.
- Risk: minimal, if the file matches your pages.
Hypothetical example: a software company with a public API links its auth guide and SDK reference in llms.txt, giving coding assistants an easy path. A local plumbing company gains far less, because nobody pastes its site into an IDE.
What moves AI search visibility is the same work covered in what GEO is: crawlable pages, clear answers, consistent entity information and real authority. If those are weak, llms.txt will not compensate. Our GEO and AI search service starts there.
How do you write a good one?
Write it for a reader with no context: one clear summary, then a short list of your best pages with a plain note on what each contains.
Link only to canonical URLs that return 200, describe content plainly, and put secondary pages under "Optional". Hypothetical example for a fictional company:
# Northwind Analytics
> Northwind Analytics makes inventory forecasting software for
> mid-size retailers. Plans start with a free tier; the API is REST
> with OAuth 2.0.
Northwind does not offer on-premise hosting. Support is email-only.
## Docs
- [Quick start](https://www.example.com/docs/quick-start.md): connect a store and run a first forecast
- [API authentication](https://www.example.com/docs/auth.md): OAuth 2.0 flow and token refresh
- [Forecast methods](https://www.example.com/docs/methods.md): how the models handle seasonality
## Company
- [Pricing](https://www.example.com/pricing.md): plan limits and what each tier includes
- [Security](https://www.example.com/security.md): data retention and compliance details
## Optional
- [Changelog](https://www.example.com/changelog.md): release notes by month
Then check it like any page: open each link, confirm the facts match the live site, and review it when pricing or products change. A stale llms.txt quoting last year's plans is worse than none, because any tool that reads it will repeat the error.
Frequently asked questions
Does llms.txt help my site appear in ChatGPT or Google AI Overviews?
There is no public confirmation that any major AI search product uses llms.txt for ranking or citation. Google states that Google Search ignores these files, and it neither helps nor hurts visibility there. Crawlable, clear HTML pages remain what gets you cited.
Is llms.txt the same as robots.txt for AI?
No. robots.txt tells crawlers which URLs they may fetch. llms.txt is a reading list and summary; it has no allow or disallow rules and no crawler is obliged to read it.
What is llms-full.txt?
It is a community convention, used by several documentation platforms, for a single file that contains the full text of your key pages. The original proposal described similar expanded files named llms-ctx.txt and llms-ctx-full.txt. It is mainly useful for documentation sites.
Where does the file go?
At the root path, for example example.com/llms.txt, served as plain text. Most business sites need only one.