Can AI crawlers access your site?
Check if GPTBot, ClaudeBot, PerplexityBot and 12 other AI crawlers can read your site — robots.txt rules plus live firewall tests, with copy-paste fixes.
Free · No signup · We test robots.txt rules and live responses for 15 AI crawlers. Nothing is stored.
Two layers decide whether AI can crawl your site
Most guides stop at robots.txt. In practice, AI crawlers get blocked at two independent layers, and the second one is invisible unless you test it:
- robots.txt rules. A
Disallow: /underUser-agent: GPTBot(or underUser-agent: *) politely asks that bot to stay out. Well-behaved AI crawlers respect it. - Firewall / CDN layer. WAF rules and bot management reject requests by user-agent before robots.txt is even read. Cloudflare has blocked AI crawlers by default on newly onboarded zones since July 2025 — many site owners are invisible to AI search and don't know it.
This checker tests both layers: it parses your robots.txt per bot, then probes your homepage with each crawler's official user-agent and compares the response against a normal browser baseline. A mismatch means your firewall — not your robots.txt — is doing the blocking.
The AI crawler list we test (15 bots, updated monthly)
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Trains OpenAI models (ChatGPT). |
| OAI-SearchBot | OpenAI | Indexes sites for ChatGPT search results & citations. |
| ChatGPT-User | OpenAI | Fetches pages when a ChatGPT user asks about your site. |
| ClaudeBot | Anthropic | Trains Anthropic models (Claude). |
| Claude-SearchBot | Anthropic | Improves Claude's web search citations. |
| Claude-User | Anthropic | Fetches pages when a Claude user asks about your site. |
| PerplexityBot | Perplexity | Indexes sites for Perplexity search results. |
| Perplexity-User | Perplexity | Fetches pages when a Perplexity user asks about your site. |
| Google-Extended | robots.txt token only — controls Gemini training; crawling is done by Googlebot. | |
| Applebot-Extended | Apple | robots.txt token only — controls Apple AI training; crawling is done by Applebot. |
| bingbot | Microsoft | Bing index also powers ChatGPT search & Copilot answers. |
| CCBot | Common Crawl | Open web corpus used to train many LLMs. |
| meta-externalagent | Meta | Trains Meta AI models (Llama). |
| Bytespider | ByteDance | Trains ByteDance models (Doubao); known to ignore robots.txt at times. |
| Amazonbot | Amazon | Powers Alexa answers and Amazon AI features. |
robots.txt for AI crawlers: allow or block, per bot
You don't need one blanket rule. The sane default for most sites that want AI visibility: allow search and user-request bots, decide consciously about training bots.
Allow AI search but opt out of training:
# AI search & citations — keep these if you want AI visibility User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Training crawlers — block if you don't want your content in training data User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: /
Remember: a specific User-agent group overrides User-agent: * entirely for that bot — rules don't combine across groups. That's the #1 source of accidental blocking we see.
Blocked at the firewall? Check Cloudflare first
If the live test shows blocks that your robots.txt doesn't explain, and your site is behind Cloudflare: open your zone → Security → Bots / Crawl Control and review the AI-bot blocking setting (default-on for zones onboarded since 2025), plus any WAF custom rules matching user-agent strings. Allow the crawlers you want, re-run the check, and you should see green.
FAQ
How do I check if AI can crawl my site?
Two layers decide it: robots.txt rules (does your file disallow GPTBot, ClaudeBot, etc.) and the network layer (does a firewall or CDN like Cloudflare block bot user-agents before robots.txt even matters). This checker tests both — it parses your robots.txt per bot and sends live requests with each crawler's official user-agent, then compares against a normal browser baseline.
Why is my website not showing up in ChatGPT?
Usual suspects, in order: your site blocks OAI-SearchBot or GPTBot in robots.txt; Cloudflare or a WAF blocks AI crawlers at the network layer (Cloudflare does this by default on newer zones since July 2025); or your site simply isn't in Bing's index — ChatGPT search leans on Bing, so being deindexed there hides you too. Run the check to rule out the first two in seconds.
Does blocking AI crawlers hurt my SEO?
Blocking training bots (GPTBot, ClaudeBot, CCBot) does not affect Google rankings — Googlebot is separate. But blocking search-oriented bots (OAI-SearchBot, PerplexityBot, bingbot) removes you from AI search answers and citations, which is a growing traffic channel. Decide per category, not with one blanket rule.
What's the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
All three are OpenAI, doing different jobs: GPTBot collects training data; OAI-SearchBot indexes pages for ChatGPT search results and citations; ChatGPT-User fetches a page live when a user asks about it. You can allow or block each independently in robots.txt.
Is Google-Extended a real crawler?
No — Google-Extended is only a robots.txt token. Crawling is still done by Googlebot; the token just tells Google not to use your content for Gemini training. Blocking Google-Extended does not affect Google Search rankings. Applebot-Extended works the same way for Apple.
Can this checker guarantee what real crawlers see?
Not 100%. Our live tests use each crawler's official user-agent but come from our server IPs. Sites that verify bots by IP ranges (like Cloudflare's verified-bots feature) may treat the real crawlers differently. robots.txt results are deterministic; live results are a strong signal, not a guarantee.
Crawlability is step one. Step two is giving AI a curated map of your content — generate a free llms.txt or read llms.txt vs robots.txt and which AI crawlers actually read llms.txt.