llms.txt vs robots.txt
They sit in the same place — your site root — and they're both plain-text files crawlers fetch. But robots.txt and llms.txt do almost opposite jobs. robots.txt is a bouncer: it tells bots which doors are locked. llms.txt is a concierge: it hands AI a curated map of what's worth reading.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Restrict crawler access | Curate content for AI |
| Audience | All crawlers / bots | LLMs & AI assistants |
| Format | Directives (User-agent / Allow / Disallow) | Markdown (headings, links, descriptions) |
| Action | Blocks or permits paths | Recommends & summarizes pages |
| Standardized? | Yes — long-established | Proposed (Answer.AI, 2024) |
| Location | /robots.txt | /llms.txt |
| Enforced? | Honored by well-behaved bots | Advisory — no enforcement |
What each file looks like
robots.txt
User-agent: * Disallow: /admin/ Allow: / User-agent: GPTBot Allow: / Sitemap: https://example.com/sitemap.xml
llms.txt
# Example Co > Open-source analytics for indie developers. ## Docs - [Quickstart](https://example.com/docs/quickstart): install in 5 minutes - [API reference](https://example.com/docs/api): every endpoint ## Optional - [Changelog](https://example.com/changelog)
The short version
Keep robots.txt for access control. Add llms.txt to help AI understand your best content. They're complementary, not competing — most AI-ready sites ship both.
Generate a spec-compliant llms.txt for your site in one click — free, no signup.
Try the free llms.txt generator →FAQ
Does llms.txt replace robots.txt?
No. They solve different problems and you keep both. robots.txt controls which paths crawlers may access; llms.txt curates which content is most important and provides clean text for AI. One is a gatekeeper, the other is a guide.
Should llms.txt and robots.txt go in the same place?
Yes — both live at your site root: /robots.txt and /llms.txt. Neither one references the other; crawlers request them independently.
If I block a path in robots.txt, should I list it in llms.txt?
No. llms.txt is a curated index of pages you want AI to understand. If you disallow a path for crawlers, don't surface it in llms.txt — keep the two consistent.
Do AI crawlers obey robots.txt?
Major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) document that they respect robots.txt, so you can allow or disallow them there. llms.txt is advisory content, not an access-control mechanism.
Related: What is llms.txt? · How to create llms.txt · Do AI crawlers read it?