# Which AI crawlers should you allow — and which you can safely block.

Nobody decides to block ChatGPT. It happens in a robots.txt line someone added for good reasons two years ago, or a bot-management toggle nobody remembers flipping. Here's what each AI crawler actually does, which ones your citations depend on, and how to find out whether yours are getting through.

_Updated 2026-08-11 · AI Syndicate GEO team_

> Source: https://www.aisyndicate.com/blog/ai-crawlers-robots-txt/

Allow every AI agent that could link to you — the retrieval crawlers (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `bingbot`, `Applebot`) and the user-triggered fetchers (`ChatGPT-User`, `Claude-User`, `Perplexity-User`). Those are your citation pipeline. Training-only crawlers — `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot` — are a **business judgment call, not an access mistake**. And robots.txt is only the first gate: a CDN bot rule or a blanket 403 blocks engines just as hard, and far more quietly.

**Every AI answer starts with a fetch.** Before an engine can cite you, some agent has to read your page — either weeks ago into an index, or two seconds ago because a buyer asked about you by name. If that fetch fails, nothing downstream matters: not your schema, not your [llms.txt](https://www.aisyndicate.com/blog/what-is-llms-txt/), not the best content in your category.

The awkward part is that blocking AI crawlers was, briefly, the default advice. Plenty of teams added a `Disallow` for GPTBot in 2023, felt responsible about it, and never revisited it. Since then the bot list has tripled and split by function — so the same line now blocks something quite different from what it blocked then.

## Which AI crawlers are actually hitting your site?

Most operators run several agents, each with its own robots.txt token and its own job. These are the ones worth knowing by name:

- **OpenAI** — `GPTBot` (crawls for model training), `OAI-SearchBot` (builds the index behind ChatGPT's search results and links), `ChatGPT-User` (a live fetch when a person or an action asks for your page).
- **Anthropic** — `ClaudeBot` (training), `Claude-SearchBot` (search indexing), `Claude-User` (live fetch on a user's request).
- **Perplexity** — `PerplexityBot` (indexing for citations), `Perplexity-User` (live fetch inside a session).
- **Google** — `Googlebot`. There is no separate AI Overviews crawler: [AI Overviews](https://www.aisyndicate.com/blog/how-ai-overviews-choose-sources/) and AI Mode are assembled from the Search index, so Googlebot _is_ the access path. `Google-Extended` isn't a crawler at all — it's a robots token that governs whether already-fetched content may be used for Gemini apps and grounded Vertex AI responses.
- **Microsoft** — `bingbot`. Copilot answers lean on the Bing index, so Bing access is Copilot access.
- **Apple** — `Applebot` (Siri, Spotlight, Apple Intelligence surfaces) and `Applebot-Extended`, which is a training opt-out only.
- **Meta** — `meta-externalagent` for its AI crawling, `meta-externalfetcher` for links a user opens.
- **Everyone else** — `Amazonbot`, `Bytespider`, `MistralAI-User`, `cohere-ai`, xAI's crawler, `DuckDuckBot`, and `CCBot`, the Common Crawl agent whose public archive feeds model builders who never crawl you directly.

> **Treat any list — including this one — as a snapshot.** Operators rename agents, split one token into three, and publish new ones without much fanfare. OpenAI's and Anthropic's search bots both arrived _after_ the training crawlers everyone had already blocked, which is exactly how a site ends up welcoming a bot that no longer matters and refusing the one that does. Verify against the operator's own documentation before you write a rule.

## Three jobs, three very different stakes

Sorting the bots by operator tells you very little. Sorting them by _what the fetch is for_ tells you what you're risking when you block one:

### Training crawlers

They read pages that go into building and refreshing a model. The effect is diffuse and delayed: it shapes what an engine can say about you _with no browsing at all_. Blocking costs you nothing today and something hard to measure later.

### Retrieval crawlers

They build the index the engine searches at answer time and cite what they find. This is the citation pipeline. Block it and you're not "less visible" — you're ineligible to be a source.

### User-triggered fetchers

They fetch your page because a specific person asked a question it answers. That's the highest-intent request on the internet: a buyer is asking about you _right now_, and whether the page opens decides what they're told.

> **One operator's tokens do not share a rule.** A crawler obeys exactly one robots.txt group — the one that names it — so blocking `GPTBot` doesn't touch ChatGPT's search index, and allowing `GPTBot` doesn't permit `ChatGPT-User`. Almost every "we blocked AI" situation we find is partial in a way nobody intended.

## Which AI crawlers should you allow?

Decide by class, not by brand. Four of the five classes have an obvious answer:

| Class | What the fetch is for | Our call |
| --- | --- | --- |
| Retrieval | Builds the index answers get assembled from — `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `bingbot`, `Applebot` | Always allow. This is the citation pipeline. |
| User-triggered | Opens your page mid-conversation because someone asked — `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `meta-externalfetcher`, `MistralAI-User` | Always allow. Blocking these is self-harm at the moment of highest intent. |
| Training | Feeds model pretraining and refreshes — `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `meta-externalagent`, `CCBot` | Judgment call. Allow if you want engines to know you unprompted; block if the corpus is the product you sell. |
| Previews & ads | Renders link cards and checks ad landing pages — `facebookexternalhit`, `AdsBot-Google` | Allow. Blocking these breaks your own shares and campaigns. |
| Volume scrapers | Crawls hard, cites rarely, and may ignore your rules anyway | Rate-limit at the edge. A polite request is the wrong tool for an impolite bot. |

_One rule of thumb covers most of it: never block the bot that could link to you._

The training row is a real decision, and reasonable people land on both sides. If your content _is_ the licensable asset — an archive, a research library, original journalism — then blocking training crawlers while allowing retrieval is a coherent position: you keep your citations and withhold the corpus. If your content exists to sell something else, blocking training mostly means the model has less basis to name you in the many answers where it never browses at all.

What blocking a _retrieval_ crawler does is worse than it sounds, because it doesn't remove you from the answer. The engine still answers the question about you — now from directories, forums, and competitor comparison pages you don't control. [Mentioned, wrong, and uncredited](https://www.aisyndicate.com/blog/why-your-brand-isnt-in-chatgpt/) is the worst of the available outcomes.

## How do you write the robots.txt?

Four mechanics do most of the work:

- **Name the agents you care about.** A crawler applies the most specific group matching its token and ignores every other group. Anything you don't name falls back to `User-agent: *` — which is how a legacy blanket rule quietly catches next year's search bot.
- **Group agents that share a policy.** Several `User-agent:` lines can stack in front of one block of rules, so you don't need a copy of the same policy nine times.
- **Keep endpoints and dead ends out.** API routes, internal search results, faceted URL sprawl, staging. That's crawl-budget hygiene, not defense.
- **Point at your machine-readable companions.** A `Sitemap:` directive, and a comment pointing to [llms.txt](https://www.aisyndicate.com/blog/what-is-llms-txt/) for engines that look for one.

```
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /api/

User-agent: *
Allow: /
Disallow: /api/
```

That's the shape. The policy itself — which agents you name, which paths you close, which exceptions your stack needs — depends on what you publish and what you sell, and it's the part that goes wrong quietly. A file that looks tidy can still be refusing the one crawler your citations run through.

Two limits worth stating plainly. First, **robots.txt is a request, not access control** — the major operators publish their tokens and say they honor it, but compliance is voluntary and unenforceable. Anything that genuinely must not be read belongs behind authentication. Second, **a `Disallow` today doesn't unlearn yesterday**: a model trained on your pages last year still carries them, so blocking is a decision about future crawls only.

> Ours is public if you want a worked example: [aisyndicate.com/robots.txt](https://www.aisyndicate.com/robots.txt) allows every AI agent we know of and disallows `/api/`, because those routes are handlers rather than pages. We also publish a [crawler policy for our own bot](https://www.aisyndicate.com/bot/) — if we ask other sites to let us read them, we owe them the same clarity.

## robots.txt is only the first gate

In practice, robots.txt is usually _not_ the thing stopping the fetch. Ordered roughly by how often we find each one:

1. **Bot management at the CDN.** Managed rule sets increasingly block "AI bots" as a category, sometimes on by default. robots.txt says allow, the edge returns 403, and nobody sees it because nobody's watching the edge logs for a bot.
2. **A blanket 403 or 429 for unfamiliar agents** at the origin — a security plugin, an aggressive rate limiter, an old WAF rule matching anything that isn't a browser.
3. **Client-side rendering.** The agent gets a 200 and an empty shell because the content arrives by JavaScript. Don't assume AI fetchers execute your app the way Googlebot does.
4. **Interstitials.** Cookie walls, age gates, human-verification checks, login walls. A wall is a block with better manners.
5. **Snippet directives.** `noindex` takes you out entirely; `nosnippet`, `max-snippet:0`, and `data-nosnippet` tell Google not to quote the exact passage you most want quoted.
6. **Geo and IP rules** that reject datacenter ranges wholesale — including the ones AI engines fetch from.

> **Report what you observed, not what you assume.** A failed fetch from your laptop is evidence of one failed fetch — not proof the site is unreachable for everyone, and definitely not a score of zero. "We couldn't read this page as `OAI-SearchBot`; the edge returned 403" is a finding you can act on. "This page has no content" is a measurement you didn't take, written up as a fact.

Checking all six by hand, per page, is the part teams skip — so it's the part we automated in [AI Access](https://www.aisyndicate.com/ai-access/), alongside the [12-point GEO audit](https://www.aisyndicate.com/blog/geo-audit-checklist/)'s content and structure checks.

## So how do you know whether you're blocked?

Not from your browser, and not from reading robots.txt. Those tell you what you _intended_. Knowing what AI engines actually get means checking four things at once, per agent and per page:

- **The response each agent really receives.** Same URL, different user-agent string, frequently a different answer.
- **Whether the body is a page or a shell.** A 200 that returns almost nothing is a block wearing a success code.
- **The edge and the origin separately.** A welcoming server behind an unwelcoming CDN reads as "allowed" everywhere except where it counts.
- **What the page permits once it's read** — the index and snippet directives that decide whether your best paragraph can be quoted at all.

And then the part no one-time check covers: **access breaks silently.** A CDN rule ships, a plugin updates, an operator adds a token, and a fetch that worked last month stops working. Nobody raises a flag, because a bot that stops arriving doesn't file a complaint — your citations just quietly thin out, and the first honest signal is a scan that says you're no longer being named.

> **Don't guess. Check, then keep checking.** Start with the free audit to see where you actually stand across the AI engines your buyers use. Then [AI Pulse](https://www.aisyndicate.com/products/pulse/) keeps it honest at $499/month: one scan across all 10 engines every month, a single GEO score, what changed since last cycle, and a ranked list of fixes — so a new block shows up in the next scan instead of a quarter later.

> Access isn't visibility. But no access is a guaranteed absence — and it's the one problem in AI search you can be certain about, if you bother to look.

## FAQ

### Should you block AI crawlers?

It depends on the crawler's job. If you want AI engines to cite and recommend you, allow the retrieval crawlers and the user-triggered fetchers — `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `bingbot`, `Applebot`, `ChatGPT-User`, `Claude-User`, `Perplexity-User` — because those are the agents that make your page a linkable source. Training-only crawlers such as `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended` and `CCBot` are a business decision: publishers who license their archive often block them, while brands that want to be named in AI answers usually allow them.

### Does blocking GPTBot remove your brand from ChatGPT?

No. GPTBot is OpenAI's crawler for model training, and blocking it neither removes what a model already learned nor governs ChatGPT's search results. ChatGPT's linked, cited answers come from a separate index built by `OAI-SearchBot`, plus real-time fetches by `ChatGPT-User`. Each token obeys only the robots.txt group that names it, so blocking one doesn't block the others — and allowing one doesn't allow the others.

### Which robots.txt user agents do the major AI engines use?

OpenAI publishes `GPTBot` (training), `OAI-SearchBot` (search index) and `ChatGPT-User` (user-triggered fetch). Anthropic publishes `ClaudeBot`, `Claude-SearchBot` and `Claude-User`. Perplexity publishes `PerplexityBot` and `Perplexity-User`. Google uses `Googlebot` for Search — which is what AI Overviews and AI Mode are built on — plus the `Google-Extended` token for Gemini and Vertex AI grounding. Microsoft's Copilot leans on the Bing index crawled by `bingbot`. Apple uses `Applebot` and `Applebot-Extended`, Meta uses `meta-externalagent` and `meta-externalfetcher`, and Common Crawl's `CCBot` feeds datasets many other model builders use. Operators add and rename tokens regularly, so re-check their published docs rather than trusting a copied list.

### Does Google-Extended affect AI Overviews?

No. `Google-Extended` is not a crawler; it's a robots.txt token controlling whether content Googlebot already fetched may be used for Gemini apps and grounded responses in Vertex AI. Google documents that it doesn't affect inclusion or ranking in Google Search, and AI Overviews are generated from the Search index. The controls that do limit what AI Overviews can quote are the snippet directives — `nosnippet`, `max-snippet` and `data-nosnippet` — and blocking Googlebot itself, which removes you from Search and AI Overviews together.

### Do AI crawlers respect robots.txt?

The major operators publish their user-agent tokens and state that their crawlers follow robots.txt, but compliance is voluntary and unenforceable — robots.txt is a request, not access control. Policies also differ for user-triggered fetches, where some operators treat a page a person explicitly asked for differently from automated crawling. Anything that genuinely must not be read needs authentication or an edge rule, not a `Disallow` line.

### How do you check whether your site is blocking AI crawlers?

Reading your robots.txt isn't enough — it tells you what you intended, not what engines receive. A real check covers four things per agent and per page: the response each AI user agent actually gets, whether the returned body is a full page or an empty shell, the edge and the origin separately, and the index and snippet directives that decide whether your text can be quoted at all. It also has to repeat, because access breaks silently when a CDN rule ships or an operator adds a new token. A monitored scan like [AI Pulse](https://www.aisyndicate.com/products/pulse/) re-runs that check every month across all 10 engines and tells you what changed. Either way, report what you observed rather than what you assume: a failed fetch from one client is evidence of a failed fetch, not proof the site is unreachable for everyone.

## Key takeaways

- Sort AI crawlers by job, not by brand: **training, retrieval, and user-triggered** fetches carry completely different stakes.
- **Never block the bot that could link to you.** Retrieval crawlers and user-triggered fetchers are the citation pipeline; blocking them doesn't remove you from answers, it removes your page as the source.
- Each token obeys only **the robots.txt group that names it** — so blocking `GPTBot` leaves ChatGPT's search index untouched, and a legacy `*` rule silently catches every new agent.
- robots.txt is the first gate, not the only one: **CDN bot rules, blanket 403s, JavaScript-only pages, and snippet directives** block engines more often — and they break silently, so [check per agent on a cadence](https://www.aisyndicate.com/products/pulse/) instead of assuming.