What Is llms.txt?
llms.txt is an emerging web standard (originating in 2024) that lets website owners provide curated guidance to AI crawlers about which content is most representative, citable, and authoritative. It is conceptually similar to robots.txt for AI consumption.
Frequently Asked Questions
What is llms.txt for?
It tells AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) which pages on your site are most authoritative, how to interpret your content, and which content you want emphasized in citations. Most major AI engines now respect llms.txt content.
Where do I put llms.txt?
In the root of your domain, like /robots.txt. So promptolis.com/llms.txt. The file is plain markdown text.
What goes in llms.txt?
Typical structure: (1) site name and one-sentence description, (2) brief context about what the site is, (3) curated list of important pages with one-line descriptions, (4) optional notes for AI crawlers about how to interpret content. Promptolis maintains a thorough llms.txt at promptolis.com/llms.txt.
Is llms.txt required?
No, it is voluntary. But most major AI engines (OpenAI, Anthropic, Google, Perplexity) now check for it and use it when present. It is becoming a de-facto standard.
What is the difference between llms.txt and robots.txt?
robots.txt tells crawlers which pages they can or cannot access (gatekeeping). llms.txt tells AI crawlers how to interpret and prioritize content (curation). Both can coexist. robots.txt is for permissions; llms.txt is for guidance.
How does llms.txt help AEO?
It increases the chance that AI engines cite your authoritative pages rather than less-relevant pages on your site. It is one of the highest-leverage AEO/GEO interventions because it directly speaks to the engines you want to influence.
Related Resources
Get new Originals every Friday
2-3 hand-crafted Originals per week. No spam, no upsells, unsubscribe in 1 click.