aicrawlable

How to create an llms.txt file

There are three ways to add llms.txt to your website — generate it, write it, or template it. All produce the same kind of file: a Markdown map of your site that AI systems can read.

1. Option 1 — Generate it automatically (fastest)

Paste your URL into a generator. It crawls your site (via sitemap.xml or homepage links), lists your pages, and writes spec-compliant Markdown for you. You pick which pages to keep and fix descriptions before exporting.

Generate llms.txt now — free →

2. Option 2 — Write it by hand

The spec is just Markdown. The only required line is the H1 title:

# Your Project Name

> One-sentence summary of what your site is.

Optional intro paragraph (no headings here).

## Docs
- [Quickstart](https://you.com/docs/start): get running in 5 minutes
- [API](https://you.com/docs/api): full endpoint reference

## Optional
- [Changelog](https://you.com/changelog)

The ## Optional section is special: those links may be skipped when an AI's context window is tight. Order matters — most important pages first.

3. Option 3 — Add llms-full.txt too

llms.txt is the index; llms-full.txt embeds the actual page content into one file so an LLM can ingest your whole site in a single fetch. If you maintain docs or want a ready-made context file for your own RAG/chatbot, generate both at once.

4. Final step — Upload & verify

Put the file at your site root and open https://yourdomain.com/llms.txt. If you see your Markdown rendered as plain text, you're done. No plugin, no build step required.

FAQ

Where do I put the llms.txt file?

At your domain root, so it's reachable at https://yourdomain.com/llms.txt — exactly like robots.txt. For static sites drop it in your public/ or root folder; for a CDN, upload it as a root asset.

How long should llms.txt be?

Keep the index (llms.txt) short — your most important pages with one-line descriptions. If you also want the full text of those pages in one file for AI ingestion, generate a separate llms-full.txt.

Do I need to regenerate it when my site changes?

Yes — llms.txt is a snapshot. Regenerate it when you add major pages or restructure your docs. It's a static file, so there's no automatic syncing.

How do I verify it works?

Visit https://yourdomain.com/llms.txt in a browser — you should see plain Markdown, served with a text/plain or text/markdown content type, not your 404 page.

Related: What is llms.txt? · llms.txt examples · llms.txt vs robots.txt