Frequently asked questions
What is llms.txt?
llms.txt is a proposed Markdown standard placed at your site root (/llms.txt) that gives AI systems a curated index of your most important pages, with short descriptions. It helps LLMs understand your site without crawling everything.
What's the difference between llms.txt and llms-full.txt?
llms.txt is a concise index — links to key pages with descriptions. llms-full.txt embeds the actual page content into one file, so an LLM can ingest your whole site in a single fetch. aicrawlable generates both.
Do AI crawlers actually read llms.txt today?
Honestly: adoption is early. As of 2026, major crawlers like GPTBot, ClaudeBot, and PerplexityBot do not reliably request llms.txt yet, and Google has called it non-essential. Think of it as low-cost preparation for an AI-ready web, and as a ready-made context file for your own RAG, docs search, or custom AI — not a guaranteed ranking boost. We'd rather tell you the truth than oversell it.
Is it really free? Do I need to sign up?
Yes, completely free. No account, no credit card, no API key. We crawl up to 50 pages per request on the edge and store nothing.
How does aicrawlable find my pages?
It first looks for sitemap.xml (including sitemap index files). If none exists, it falls back to discovering links from your homepage. You then choose which pages to keep.
Where do I put the generated file?
Upload llms.txt to your site root so it's reachable at https://yourdomain.com/llms.txt. Do the same for llms-full.txt if you use it.
Does it work for non-WordPress sites?
Yes. Unlike many generators that are WordPress plugins, aicrawlable works with any website — static sites, Next.js, Shopify, docs platforms, anything reachable over HTTP.