Find out which AI systems are allowed to read your site — and whether any block is deliberate
or a wildcard side effect. Free, no account.
Two different things people mean by "block AI"
Blocking crawlers
Rules naming GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, CCBot or Bytespider stop those bots fetching your pages. That costs freshness and citations in AI answers.
Opting out of training
Google-Extended, anthropic-ai and Applebot-Extended are not crawlers — they are tokens that ask a company not to train models on your content. Blocking one changes nothing about search rankings or live answers.
The wildcard trap
A file with User-agent: * / Disallow: / —
staging protection that reached production, usually — blocks every AI crawler that has no explicit
group of its own. Each verdict here says whether it came from a rule naming the bot or from the
wildcard, so an accidental block is recognisable as one. Pair this with the
llms.txt checker
to confirm the file AI systems look for is itself reachable.
Frequently asked questions
Will blocking GPTBot remove my content from ChatGPT?
Not exactly, and the distinction matters. Blocking a crawler stops future fetching — it does not delete what is already in training data, and it does not stop ChatGPT fetching a page live when a user links it (that is ChatGPT-User, a separate token). Treat robots rules as controlling future access, not as erasing the past.
What is Google-Extended?
A robots.txt token, not a crawler. It controls whether your content is used to train Google generative AI models such as Gemini. Blocking it does not affect Google Search rankings, and allowing it does not improve them. The same applies to anthropic-ai and Applebot-Extended: training opt-outs, not crawl controls.
Does blocking AI crawlers hurt my Google or Bing rankings?
No — search crawlers use different user-agent tokens (Googlebot, Bingbot), and a rule naming an AI bot only affects that bot. The exception is a wildcard: "User-agent: * / Disallow: /" blocks every bot that has no explicit group, AI crawlers included. That is the most common accidental AI block this tool finds.
Why does the tool say a bot is governed by the wildcard group?
Because your file has no group naming that bot, so the "*" group applies to it by the specification's group-selection rule. If you want different treatment for one AI bot — allowing PerplexityBot while blocking others, for example — it needs its own User-agent group.
Which bots matter for being cited in AI answers?
The retrieval side, not the training side: OAI-SearchBot and PerplexityBot crawl for answers with citations, and ChatGPT-User fetches pages users link in conversation. GPTBot and ClaudeBot matter for longer-term freshness. Training-control tokens do not affect citations either way.
Should I block all AI crawlers?
It depends what you are protecting against. Blocking crawlers costs visibility in AI answers and citations; allowing them costs inclusion in future training data. The middle ground is leaving crawlers allowed while blocking the training-control tokens (Google-Extended, anthropic-ai, Applebot-Extended) — though not every AI company honours every token, so read each one as a request rather than a lock.