Do AI crawlers actually read llms.txt?
Short, honest answer: not reliably, not yet. The llms.txt standard is young (proposed by Answer.AI in 2024), and as of 2026 no major AI crawler has documented that it fetches third-party /llms.txt during routine crawling. We'd rather tell you that than sell you a ranking miracle.
Where each player stands today
| Crawler / system | Reads llms.txt? |
|---|---|
| GPTBot (OpenAI) | No public evidence; honors robots.txt |
| ClaudeBot (Anthropic) | Ships its own llms.txt; no confirmation it reads yours |
| PerplexityBot | No documented support |
| Googlebot / Google-Extended | Google says it doesn't rely on llms.txt |
| Your own RAG / chatbot | Yes — if you point it at the file |
So is it pointless? No.
The value of llms.txt today isn't crawler magic — it's two concrete things:
- Cheap future-proofing. A static Markdown file costs you minutes. If/when adoption grows, you're already set up — no scramble later.
- llms-full.txt is useful right now. It bundles your site's content into one clean file — ideal as a context source for your own AI features, or for anyone who pastes it into ChatGPT/Claude to ask questions about your product.
Think of it as prep for an AI-ready web, not a guaranteed traffic lever. That's the honest framing — and it's why our generator is free.
Want the file ready for the day it matters? Generate llms.txt + llms-full.txt now.
Generate both files — free →FAQ
Does ChatGPT / GPTBot read llms.txt?
As of 2026, OpenAI has not documented GPTBot requesting /llms.txt, and there's no reliable public evidence it ingests the file. GPTBot does honor robots.txt for access control.
Does Claude / ClaudeBot read llms.txt?
Anthropic publishes llms.txt files for its own docs, which signals support for the idea — but there's no public confirmation that ClaudeBot fetches third-party llms.txt during crawling. Treat it as advisory, not guaranteed.
Does Google use llms.txt for AI Overviews?
Google has publicly said llms.txt is not something it relies on, and Google-Extended/Googlebot behavior is driven by your normal pages, sitemap, and robots.txt — not llms.txt.
If crawlers don't read it, why add llms.txt at all?
Two reasons. First, it's low-cost preparation: when adoption grows, your file is already there. Second, llms-full.txt is an immediately useful artifact today — a clean, single-file context source for your own RAG, docs search, or chatbot, and for users who manually paste it into an AI.
Related: What is llms.txt? · llms.txt vs robots.txt