Can AI engines read your site?

Citation crawlers vs training crawlers

Most robots.txt advice treats "AI bots" as one group. They are two, and blocking one does not block the other:

• Citation crawlers fetch a page live to answer a user's question and cite it. Block these and the engine genuinely cannot quote you. These are OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot.
• Training crawlers collect content for model training. Blocking these keeps you out of training sets but does not stop citations. These are GPTBot, ClaudeBot, CCBot and Google-Extended.

This distinction is where most sites get it wrong in both directions — either blocking training crawlers and assuming they have opted out of AI answers entirely, or blocking everything and quietly removing themselves from AI search without noticing.

Why we split the verdict in two

We built this check after one of our own sites tripped it. Its robots.txt disallowed GPTBot and ClaudeBot, and our first instinct was the obvious one: that is why ChatGPT and Claude never cite it.

That was wrong. Both are training crawlers. The citation crawlers were allowed the whole time, so the block explained nothing — the site simply was not being cited on merit. Every tool that collapses "AI bots" into one number would have sold us a fix for a problem we did not have, and left the real one untouched. That is why this check reports the two tiers separately, and why a training-only block is reported as deliberate rather than dressed up as an emergency.

Common questions

Does blocking GPTBot stop ChatGPT from citing my site?

No. GPTBot is OpenAI's training crawler — it collects content that may be used to train models. ChatGPT's search features fetch pages with OAI-SearchBot and ChatGPT-User instead. Blocking GPTBot keeps your content out of training data but leaves you fully citable in ChatGPT search.

Which crawlers actually matter for AI citations?

The citation crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), and PerplexityBot (Perplexity). These fetch a page live to answer a question and cite it. If your robots.txt disallows them, those engines cannot quote your pages.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls whether your content is used for Gemini and model training. Google AI Overviews are built from the regular Googlebot index, so blocking Google-Extended does not remove you from AI Overviews.

What about Perplexity-User?

Perplexity-User is a user-initiated fetch, and Perplexity documents that it disregards robots.txt. Disallowing it in robots.txt therefore has no practical effect. We report its status but never count it as a block.

Is being crawlable enough to get cited?

No — it is the floor, not the goal. Crawler access only makes you eligible. Whether an engine actually quotes you depends on authority signals, earned third-party mentions, and how quotable your page structure is.

This check reads robots.txt only. It does not verify server-level blocks (firewall rules, WAF, or Cloudflare bot rules can still stop a crawler that robots.txt allows). Crawler names verified against OpenAI, Anthropic and Perplexity documentation, August 2026.

Crawlable but still not cited? Run a free citation scan to see which engines quote you today, or read the GEO field guide.