A reference for anyone who wants ChatGPT, Claude or Perplexity to be able to cite their site.
Every major AI vendor runs two different crawlers, and blocking them means two completely different things.
One crawler builds the search index that the assistant reads when it answers a question. Block that one and your site cannot be cited, no matter how good your content is.
The other crawler collects text for model training. Blocking it is a normal business decision and it does not affect whether you get cited today.
Most guides you will find treat the two kinds as one thing. That difference decides whether a customer asking ChatGPT for a recommendation ever hears your name.
OAI-SearchBot builds the index behind ChatGPT Search. The crawler named ChatGPT-User will fetch a page when someone follows a link inside a chat. Claude-SearchBot and Claude-User do the same for Claude. PerplexityBot and Perplexity-User do the same for Perplexity.
GPTBot collects text for OpenAI model training. ClaudeBot and anthropic-ai do the same for Anthropic. Google-Extended controls Gemini training, and it has no effect on Google Search or on AI Overviews. Applebot-Extended, CCBot and Bytespider round out the list.
The practical consequence is short. If a site blocks GPTBot, nothing is broken. If a site blocks OAI-SearchBot, the site has removed itself from ChatGPT answers, and almost nobody who did it knows.
Each of these came out of parsing real files rather than reading the specification.
Two or more User-agent lines in a row belong to a single group. Site builders write this constantly: one User-agent line for GPTBot, another for OAI-SearchBot directly below it, then a single Disallow. Both bots are blocked. A parser that reads only the last name will miss it.
A named group beats the wildcard. If User-agent star says Disallow slash, and a separate group for OAI-SearchBot says Allow slash, then ChatGPT is allowed. Reading these in the wrong order produces a false alarm.
Inside one group, the longest matching rule wins. When two rules are the same length, Allow wins. The standard says so, and it leaves no room for preference.
An empty Disallow line means everything is permitted. It is not a rule and should not be counted as one.
A few hosting providers will serve HTML where the robots.txt file should be. They answer any unknown path with the homepage and a 200 code. Parsing that produces invented verdicts, so check the first few hundred characters for an html tag and treat the file as missing.
No file means everything is open. A 404 and a network error are treated the same way.
Run this and read what comes back.
curl -s https://yourdomain.com/robots.txt | grep -iA2 "OAI-SearchBot\|Claude-SearchBot\|PerplexityBot"An empty result is good news when no file exists at all, and bad news when the file exists and blocks those names.
We build askChat Studio, which measures whether AI assistants name a local business when customers ask for a recommendation. Crawler access is the first thing we check, because everything else is pointless when the door is closed. The reference is here because we needed it ourselves and found no clear version of it anywhere.