Should You Allow GPTBot, ClaudeBot and PerplexityBot?

If you want to be cited in AI answers, you need to allow the search crawlers that feed them — chiefly OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, and Claude-SearchBot for Claude. Blocking these, often accidentally, removes you from those engines’ answers entirely. Separately, you can choose whether to allow the training crawlers (GPTBot, ClaudeBot, Google-Extended) that feed model training. The two decisions are independent, and confusing them is one of the most common and costly AI-visibility mistakes. This post explains which bot is which and how to set your robots.txt correctly.

Should I allow AI crawlers on my site?

If you want AI visibility, yes — you should allow the search and retrieval crawlers that power AI answers, because blocking them removes you from those engines’ citations. OpenAI’s documentation is explicit: sites opted out of OAI-SearchBot will not appear in ChatGPT search answers. The same logic applies to PerplexityBot and Claude-SearchBot. If your buyers ask AI engines about your category, blocking these bots means your brand can’t be named.

The training crawlers are a separate choice. Allowing them can build long-term familiarity with your brand inside the models themselves; blocking them keeps your content out of training data. Neither choice affects whether you’re cited in live search answers — that’s governed entirely by the search crawlers. Decide the two independently.

The three families of AI bot

AI crawlers fall into three groups, and mixing them up is where mistakes happen.

• Training crawlers gather data in bulk to train foundation models. They crawl without urgency and respect robots.txt. The main ones are GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Gemini training), and CCBot (Common Crawl). Blocking these keeps your content out of future model training but does not affect live citations.
• Search and retrieval crawlers continuously index the web to answer live user queries, and these are the ones that decide AI citations. The key names are OAI-SearchBot (ChatGPT Search), PerplexityBot (Perplexity), and Claude-SearchBot (Claude retrieval), alongside Bingbot and Googlebot, which feed multiple engines. Allow these if you want to be cited.
• Conversational agents fetch a page when a user explicitly asks an assistant to visit it — ChatGPT-User, Claude-User, Perplexity-User. These are user-directed, so they often behave more like a browser than a crawler, and robots.txt rules may not apply the way you expect.

The recommended setup for AI visibility

For most businesses that want to be cited, the sensible default is: allow all search and retrieval crawlers, and decide training-bot policy based on whether you want your content in model training. A common, visibility-friendly configuration allows both, since brand familiarity in models is usually a benefit for marketing sites. A more guarded configuration allows search bots while blocking training bots.

Here’s a visibility-first robots.txt pattern that allows the citation-critical crawlers explicitly:

# Allow AI search / retrieval (feeds citations)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /

# Allow standard search engines (feed Bing/Google-based AI)
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /

# Training crawlers — allow for brand familiarity, or block to opt out
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /

# Default
User-agent: *
Allow: /

To opt out of training while keeping citation eligibility, change the training-crawler lines to Disallow: /. The search crawlers above stay Allow: / either way — that’s what preserves your presence in AI answers.

Check your CDN, not just robots.txt

Robots.txt is only half the picture. Many sites sit behind a CDN like Cloudflare that offers a one-click “block AI scrapers” setting, and that setting can silently override your robots.txt, blocking the very search crawlers you meant to allow. This is how so many sites accidentally lock themselves out of AI citations without touching their robots.txt at all.

Audit both layers. In Cloudflare, check the Bots or AI-crawler controls and confirm you’re not blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot. Then verify against your server logs that these crawlers are actually reaching your pages. A robots.txt that says “allow” means nothing if the CDN in front of it says “block.”

A note on compliance and spoofing

Robots.txt is a polite request, not an enforced rule. Most major crawlers honour it, but some have been documented ignoring it, and any bot can spoof a user-agent string to claim it’s something it isn’t. So two cautions apply: for genuinely non-compliant scrapers, control has to happen at the server or firewall level, not in robots.txt; and to confirm a request claiming to be a given crawler is genuine, verify it against the vendor’s published IP ranges or via reverse DNS rather than trusting the user-agent alone.

Frequently asked questions

The bottom line

The rule is simple: allow the search crawlers if you want AI citations, and decide training crawlers separately based on your data policy. Set your robots.txt to explicitly allow OAI-SearchBot, PerplexityBot and Claude-SearchBot, then check your CDN isn’t quietly blocking them anyway. It’s a ten-minute audit that determines whether you’re eligible to be cited at all — the cheapest high-stakes check in AI visibility.

We audit crawler access and CDN settings as the first step of our AI Visibility service — because no content work matters if the engines can’t reach you.