AI Crawler User Agents: The Full Reference List
The full AI crawler reference: training crawlers, search crawlers and user-action fetchers, with the exact user agents. Which to allow for citation and which you can block.
If you want AI engines to cite you, you need to let their crawlers in — and that starts with knowing exactly which user agents to allow. AI crawlers fall into three groups: training crawlers (that gather data to train models), search crawlers (that fetch pages to answer live queries and feed citations), and user-action fetchers (that retrieve a page because a user asked). For AI visibility, the search crawlers matter most: block them and you vanish from those engines’ answers. This is the full reference list and what each one does.
Key takeaway
- AI crawlers come in three types: training crawlers, search/retrieval crawlers, and user-action fetchers — with different purposes.
- Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, Googlebot) feed citations — blocking them removes you from those answers.
- You can allow search crawlers while blocking training crawlers if you want citation without contributing training data.
AI crawler types and what they control
| Training crawler | Search crawler | |
|---|---|---|
| OpenAI | GPTBot — model training | OAI-SearchBot — ChatGPT Search citations |
| Anthropic | ClaudeBot / anthropic-ai | Claude-SearchBot |
| Perplexity | — | PerplexityBot |
| Google-Extended (Gemini training) | Googlebot — AI Overviews, AI Mode | |
| Microsoft | — | Bingbot — Copilot, ChatGPT search |
| Common Crawl | CCBot | — |
The three types of AI crawler
- Training crawlers gather content to train or improve AI models. Allowing them contributes your content to model training; blocking them keeps your content out of training data. They generally respect robots.txt. Examples include GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider.
- Search/retrieval crawlers fetch pages to build the indexes that power live AI search answers and citations. These are the ones that matter most for AI visibility — if they can’t reach you, you can’t be cited in those engines’ answers. Examples include OAI-SearchBot, PerplexityBot, Claude-SearchBot, plus Bingbot and Googlebot (which serve both traditional and AI search).
- User-action fetchers retrieve a specific page because a user asked the assistant to, in real time. Examples include ChatGPT-User, Claude-User and Perplexity-User. Blocking these can stop assistants from accessing your page when a user directs them to it.
The reference list
For OpenAI: GPTBot (training), OAI-SearchBot (search, feeds ChatGPT Search citations), ChatGPT-User (user-action). For Anthropic: ClaudeBot / anthropic-ai (training), Claude-SearchBot (search), Claude-User (user-action). For Perplexity: PerplexityBot (search/index), Perplexity-User (user-action). For Google: Googlebot (search, including AI Overviews and AI Mode), Google-Extended (controls Gemini training use). For Microsoft/Bing: Bingbot (search, powers Copilot and ChatGPT search retrieval). Plus Common Crawl’s CCBot (training) and others. The key distinction to keep straight is search versus training — because you may want opposite policies for each.
Search ≠ training
The distinction that matters most. Search/retrieval crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) feed live citations; training crawlers (GPTBot, ClaudeBot, CCBot) gather model-training data. You can allow one and block the other.
Source — AI crawler documentation, 2026
Allowing citation while controlling training
Because search and training crawlers are separate user agents, you can choose different policies. Want to appear in AI answers but not contribute to model training? Allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, plus Bingbot and Googlebot) while blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). This is a legitimate, common stance. The critical warning: blocking a search crawler removes you from that engine’s answers entirely — for instance, blocking OAI-SearchBot removes you from ChatGPT Search. If AI visibility is a goal, never block the search crawlers.
Verify, don’t just trust the name
User-agent strings can be spoofed, so for anything security-sensitive, verify a crawler is genuine using its published IP ranges or reverse DNS rather than trusting the name alone. For everyday allow/block decisions in robots.txt, matching by user-agent name is the practical mechanism — but be aware the identity claim isn’t proof. And check both layers: robots.txt and any CDN or security service, since a surprisingly common problem is a CDN silently blocking AI crawlers even when robots.txt allows them.
Frequently asked questions
What are the main AI search crawler user agents?
The key search/retrieval crawlers that feed AI citations are OAI-SearchBot (OpenAI/ChatGPT Search), PerplexityBot (Perplexity), Claude-SearchBot (Anthropic), plus Bingbot (which powers Copilot and ChatGPT retrieval) and Googlebot (which serves AI Overviews and AI Mode). These are the ones to allow for AI visibility — if they can’t reach your pages, you can’t be cited in those engines’ answers.
What’s the difference between training and search AI crawlers?
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) gather content to train or improve AI models. Search/retrieval crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, Googlebot) fetch pages to power live AI answers and citations. They’re separate user agents, so you can set different policies — for example allowing search crawlers for citation while blocking training crawlers to keep your content out of model training.
Can I allow AI citation but block AI training?
Yes. Because search and training crawlers are distinct user agents, you can allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, Googlebot) while blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). This lets you appear in AI answers without contributing to model training — a legitimate, common stance. Just never block the search crawlers if AI visibility is a goal.
How do I know a crawler is really who it claims?
User-agent strings can be spoofed, so for security-sensitive decisions, verify using the crawler’s published IP ranges or reverse DNS rather than trusting the name. For routine robots.txt allow/block rules, matching by user-agent name is the practical mechanism, but the identity claim isn’t proof. Also check both robots.txt and any CDN or security service, since CDNs sometimes block AI crawlers even when robots.txt allows them.
The bottom line
AI crawlers split into training, search and user-action types, and the search crawlers — OAI-SearchBot, PerplexityBot, Claude-SearchBot, Bingbot, Googlebot — are the ones that decide whether you can be cited. Allow those always; you can block training crawlers separately if you prefer. Check both robots.txt and your CDN, and verify identity when it matters. Get the access right, and you’ve cleared the first gate to AI citation.
We audit and fix AI crawler access across robots.txt and CDN so every engine can reach you. Part of our AI Visibility service.
