Which AI crawlers should you allow — and which you can safely block.

Every AI answer starts with a fetch. Before an engine can cite you, some agent has to read your page — either weeks ago into an index, or two seconds ago because a buyer asked about you by name. If that fetch fails, nothing downstream matters: not your schema, not your llms.txt, not the best content in your category.

The awkward part is that blocking AI crawlers was, briefly, the default advice. Plenty of teams added a Disallow for GPTBot in 2023, felt responsible about it, and never revisited it. Since then the bot list has grown and split by function — so the same line now blocks something quite different from what it blocked then.

Which AI crawlers are actually hitting your site?

The AI crawlers worth knowing by name belong to OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, Mistral and Common Crawl. Most operators run several agents, each with its own robots.txt token and its own job:

• OpenAI — GPTBot (crawls for model training), OAI-SearchBot (builds the index behind ChatGPT's search results and links), ChatGPT-User (a live fetch when a person or an action asks for your page), plus OAI-AdsBot, which checks pages submitted as ChatGPT ads. OpenAI
• Anthropic — ClaudeBot (training), Claude-SearchBot (search indexing), Claude-User (live fetch on a user's request). Anthropic
• Perplexity — PerplexityBot (indexing for its search results and citations; Perplexity says it is not used for model training), Perplexity-User (live fetch inside a session). Perplexity
• Google — Googlebot. There is no separate AI Overviews crawler: AI Overviews and AI Mode are assembled from the Search index, so Googlebot is the access path. Google Search Central Google-Extended isn't a crawler at all — it's a robots token that governs whether content Google crawls may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. Google
• Microsoft — bingbot. Copilot answers lean on the Bing index, so Bing access is Copilot access. Bing Webmaster Tools now reports how often your pages are cited in Copilot answers. Bing
• Apple — Applebot (search in Spotlight, Siri and Safari; Apple says its data may also help train Apple's models) and Applebot-Extended, which is a training opt-out only. Apple states: "Applebot-Extended does not crawl webpages." Apple
• Meta — meta-webindexer for Meta AI search results, meta-externalagent for AI training and direct indexing, meta-externalfetcher for links a user asks for. Meta
• Amazon — Amazonbot (may be used to train Amazon AI models), Amzn-SearchBot (search experiences such as Alexa), Amzn-User (live fetch for a user's question). Amazon
• Mistral — MistralAI-Index (search index for its Vibe assistant), MistralAI-User (live fetch), MistralAI-Training (training). Mistral
• Everyone else — DuckDuckBot and DuckAssistBot (DuckDuckGo's real-time crawler for its AI-assisted answers), DuckDuckGo and CCBot, the Common Crawl agent whose public archive feeds model builders who never crawl you directly. Smaller operators come and go; your server logs show which ones actually visit.

Three jobs, three very different stakes

Sorting the bots by operator tells you very little. Sorting them by what the fetch is for tells you what you're risking when you block one:

Which AI crawlers should you allow?

Allow every retrieval crawler and user-triggered fetcher, and treat training crawlers as a business decision. Decide by class, not by brand. Four of the five classes have an obvious answer:

The training row is a real decision, and reasonable people land on both sides. If your content is the licensable asset — an archive, a research library, original journalism — then blocking training crawlers while allowing retrieval is a coherent position: you keep your citations and withhold the corpus. If your content exists to sell something else, blocking training mostly means the model has less basis to name you in the many answers where it never browses at all.

What blocking a retrieval crawler does is worse than it sounds, because it doesn't remove you from the answer. The engine still answers the question about you — now from directories, forums, and competitor comparison pages you don't control. Mentioned, wrong, and uncredited is the worst of the available outcomes.

How do you write the robots.txt?

Give the AI agents you care about their own named groups, and let User-agent: * cover everything else. Four mechanics do most of the work:

• Name the agents you care about. A crawler applies the most specific group matching its token and ignores every other group. Anything you don't name falls back to User-agent: * — which is how a legacy blanket rule quietly catches next year's search bot. The rules are set out in RFC 9309, the robots.txt standard.
• Group agents that share a policy. Several User-agent: lines can stack in front of one block of rules, so you don't need a copy of the same policy nine times.
• Keep endpoints and dead ends out. API routes, internal search results, faceted URL sprawl, staging. That's crawl-budget hygiene, not defense.
• Point at your machine-readable companions. A Sitemap: directive, and a comment pointing to llms.txt for engines that look for one.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /api/

User-agent: *
Allow: /
Disallow: /api/

That's the shape. The policy itself — which agents you name, which paths you close, which exceptions your stack needs — depends on what you publish and what you sell, and it's the part that goes wrong quietly. A file that looks tidy can still be refusing the one crawler your citations run through.

Two limits worth stating plainly. First, robots.txt is a request, not access control — the major operators publish their tokens and say they honor it, but compliance is voluntary and unenforceable. The standard itself says: "These rules are not a form of access authorization." RFC 9309 User-triggered fetchers are looser still: OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules," because a person asked for the page. Anything that genuinely must not be read belongs behind authentication. Second, a Disallow today doesn't unlearn yesterday: a model trained on your pages last year still carries them, so blocking is a decision about future crawls only.

robots.txt is only the first gate

In practice, robots.txt is usually not the thing stopping the fetch. Ordered roughly by how often we find each one:

1. Bot management at the CDN. Managed rule sets increasingly block "AI bots" as a category, sometimes on by default. robots.txt says allow, the edge returns 403, and nobody sees it because nobody's watching the edge logs for a bot.
2. A blanket 403 or 429 for unfamiliar agents at the origin — a security plugin, an aggressive rate limiter, an old WAF rule matching anything that isn't a browser.
3. Client-side rendering. The agent gets a 200 and an empty shell because the content arrives by JavaScript. Don't assume AI fetchers execute your app the way Googlebot does.
4. Interstitials. Cookie walls, age gates, human-verification checks, login walls. A wall is a block with better manners.
5. Snippet directives. noindex takes you out entirely; nosnippet, max-snippet:0, and data-nosnippet tell Google not to quote the exact passage you most want quoted. Google
6. Geo and IP rules that reject datacenter ranges wholesale — including the ones AI engines fetch from.

Checking all six by hand, per page, is the part teams skip — so it's the part we automated in AI Access, alongside the 12-point GEO audit's content and structure checks.

So how do you know whether you're blocked?

You know by testing what each AI agent actually receives, page by page. Not from your browser, and not from reading robots.txt. Those tell you what you intended. Knowing what AI engines actually get means checking four things at once, per agent and per page:

• The response each agent really receives. Same URL, different user-agent string, frequently a different answer.
• Whether the body is a page or a shell. A 200 that returns almost nothing is a block wearing a success code.
• The edge and the origin separately. A welcoming server behind an unwelcoming CDN reads as "allowed" everywhere except where it counts.
• What the page permits once it's read — the index and snippet directives that decide whether your best paragraph can be quoted at all.

And then the part no one-time check covers: access breaks silently. A CDN rule ships, a plugin updates, an operator adds a token, and a fetch that worked last month stops working. Nobody raises a flag, because a bot that stops arriving doesn't file a complaint — your citations just quietly thin out, and the first honest signal is a scan that says you're no longer being named.

Access isn't visibility. But no access is a guaranteed absence — and it's the one problem in AI search you can be certain about, if you bother to look.

Sources

1. Overview of OpenAI Crawlers — OpenAI. Read Sep 30, 2026.
2. Does Anthropic crawl data from the web, and how can site owners block the crawler? — Claude Help Center (Anthropic). Read Sep 30, 2026.
3. Perplexity Crawlers — Perplexity. Read Sep 30, 2026.
4. Google's common crawlers — Google for Developers. Read Sep 30, 2026.
5. AI Features and Your Website — Google Search Central. Read Sep 30, 2026.
6. Introducing AI Performance in Bing Webmaster Tools Public Preview — Bing Webmaster Blog. Read Sep 30, 2026.
7. About Applebot — Apple Support. Read Sep 30, 2026.
8. Meta Web Crawlers — Meta for Developers. Read Sep 30, 2026.
9. About AmazonBot — Amazon Developer. Read Sep 30, 2026.
10. Mistral crawlers — Mistral Docs. Read Sep 30, 2026.
11. Is DuckAssistBot related to DuckDuckGo? — DuckDuckGo Help Pages. Read Sep 30, 2026.
12. CCBot — Common Crawl. Read Sep 30, 2026.
13. RFC 9309: Robots Exclusion Protocol — IETF (RFC Editor). Read Sep 30, 2026.