Glossary · SEO & AI search
What is llms.txt?
llms.txt is a proposed standard for a Markdown file served at a website's root (/llms.txt) that gives large language models and AI tools a concise, curated summary of the site and links to its most useful content.
It was proposed by Jeremy Howard of Answer.AI in September 2024, as a companion to robots.txt and sitemaps aimed at language models rather than crawlers.
Also called: llms-full.txt
Updated
llms.txt format
# VisitTrack > Cookie-free web analytics with revenue attribution. ## Docs - [Install](https://example.com/docs/install): One script tag. - [Cookieless mode](https://example.com/docs/cookieless): How it works. ## Optional - [Blog](https://example.com/blog)
The spec asks for an H1 with the site or project name, a blockquote summary, optional free-text detail, then H2 sections of Markdown link lists with short descriptions. A section named "Optional" marks links that can be skipped when context is short. Some sites also publish llms-full.txt with the full text of their docs in one file.
Does llms.txt actually work?
Adoption has been strongest among developer tools and documentation sites, and some AI coding assistants and IDE tools read these files when pointed at them. But no major search engine has committed to using it for ranking or AI answers; Google representatives have said Google Search doesn't use it. Treat it as cheap, harmless documentation for AI tools — not a ranking factor and not a substitute for crawlable HTML.
llms.txt example use
A developer asks an AI coding assistant to integrate your API and gives it your docs URL. With an llms-full.txt, the assistant can read your whole reference in one request instead of crawling dozens of pages, and is more likely to use current endpoints. That's where llms.txt earns its keep today.
llms.txt pitfalls
- Letting it go stale — an outdated file is worse than none.
- Listing everything; the point is curation.
- Assuming it controls access. It doesn't — use robots.txt for that.
- Expecting traffic from it.
llms.txt and VisitTrack
VisitTrack publishes its own /llms.txt, generated from the same registry as its sitemap so it never drifts. Create one for your site with the free llms.txt generator. To see whether AI tools actually fetch it — or your other pages — VisitTrack's AI crawler tracking shows each crawler's requests by path.
Frequently asked questions
Is llms.txt an official standard?
No. It's a community proposal from September 2024, not a standard from a standards body, and support varies by tool.
Does Google use llms.txt?
Google has said Google Search doesn't use llms.txt. Its AI features rely on its normal crawling and indexing, controlled through robots.txt and tokens like Google-Extended.
Related terms
- robots.txtrobots.txt is a plain-text file at the root of a website (/robots.txt) that tells crawlers which URLs they are allowed or not allowed to request, using the Robots Exclusion Protocol, standardized as RFC 9309 in 2022.
- Generative engine optimization (GEO)Generative engine optimization (GEO) is the practice of shaping a website's content and technical setup so that AI-powered answer engines — ChatGPT search, Perplexity, Google AI Overviews, Claude, Copilot — retrieve it, cite it and mention it in their generated answers.
- AI crawlersAI crawlers are automated bots operated by AI companies that fetch web pages — either to collect training data for models, to index content for AI search, or live, to answer a user's question in an assistant like ChatGPT, Claude or Perplexity.
- Structured dataStructured data is machine-readable markup added to a web page — usually Schema.org vocabulary written as JSON-LD — that describes what the page contains (an article, product, organization, FAQ, defined term) so search engines and other systems can understand it precisely.
- Answer engine optimization (AEO)Answer engine optimization (AEO) is the practice of structuring content so that search engines and assistants can use it as the direct answer to a question — in featured snippets, People Also Ask boxes, voice assistants and AI-generated answers.
Tools and guides
See which channels actually bring paying customers
VisitTrack is cookie-free analytics with revenue attribution built in. One script tag, no consent banner, live in two minutes. 14 days free, no card required.