Do AI crawlers actually read llms.txt?
Short, honest answer: not reliably, not yet. The llms.txt standard is young (proposed by Answer.AI in 2024), and as of 2026 no major AI crawler has documented that it fetches third-party /llms.txt during routine crawling. We'd rather tell you that than sell you a ranking miracle.
What crawlers do honor is robots.txt and the network layer — so before worrying about llms.txt, it's worth confirming AI bots can even reach your pages. You can check whether GPTBot, ClaudeBot and others can crawl your site in one click.
Where each player stands today
| Crawler / system | Reads llms.txt? |
|---|---|
| GPTBot (OpenAI) | No public evidence; honors robots.txt |
| ClaudeBot (Anthropic) | Ships its own llms.txt; no confirmation it reads yours |
| PerplexityBot | No documented support |
| Googlebot / Google-Extended | Google says it doesn't rely on llms.txt |
| Your own RAG / chatbot | Yes — if you point it at the file |
So is it pointless? No.
The value of llms.txt today isn't crawler magic — it's two concrete things:
- Cheap future-proofing. A static Markdown file costs you minutes. If/when adoption grows, you're already set up — no scramble later.
- llms-full.txt is useful right now. It bundles your site's content into one clean file — ideal as a context source for your own AI features, or for anyone who pastes it into ChatGPT/Claude to ask questions about your product.
Think of it as prep for an AI-ready web, not a guaranteed traffic lever. That's the honest framing — and it's why our generator is free.
Want the file ready for the day it matters? Generate llms.txt + llms-full.txt now.
Generate both files — free →FAQ
Does ChatGPT / GPTBot read llms.txt?
As of 2026, OpenAI has not documented GPTBot requesting /llms.txt, and there's no reliable public evidence it ingests the file. GPTBot does honor robots.txt for access control.
Does Claude / ClaudeBot read llms.txt?
Anthropic publishes llms.txt files for its own docs, which signals support for the idea — but there's no public confirmation that ClaudeBot fetches third-party llms.txt during crawling. Treat it as advisory, not guaranteed.
Does Google use llms.txt for AI Overviews?
Google has publicly said llms.txt is not something it relies on, and Google-Extended/Googlebot behavior is driven by your normal pages, sitemap, and robots.txt — not llms.txt.
If crawlers don't read it, why add llms.txt at all?
Two reasons. First, it's low-cost preparation: when adoption grows, your file is already there. Second, llms-full.txt is an immediately useful artifact today — a clean, single-file context source for your own RAG, docs search, or chatbot, and for users who manually paste it into an AI.
Related: What is llms.txt? · llms.txt vs robots.txt · Is llms.txt worth it? · Does Google use llms.txt? · Check if AI crawlers can reach your site