What are AI crawlers?
What each major AI bot does, according to the company that runs it — and why the difference decides whether you can be cited.
AI crawlers are the bots AI companies send to read web pages. They do three different jobs: collecting pages that may be used to train models, building the search index an AI engine cites from, and fetching a page live because a user asked. You control most of them in your robots.txt file, one bot name at a time.
Which AI crawlers should I know by name?
Each line below says only what the company's own documentation says. robots.txt is the plain-text file at the root of your site that tells bots what they may read.
| Bot | Company | What it does, per the company |
|---|---|---|
GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models. OpenAI |
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features. Sites that opt out are not shown in ChatGPT search answers. OpenAI |
ChatGPT-User | OpenAI | Visits a page when a user's question in ChatGPT or a Custom GPT calls for it. OpenAI says robots.txt rules may not apply, because a user started it. OpenAI |
ClaudeBot | Anthropic | Collects web content that could be used to train Anthropic's models. Blocking it signals your future pages should be left out of training. Anthropic |
Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results. Blocking it may reduce your visibility in those results. Anthropic |
Claude-User | Anthropic | Fetches a page when a person asks Claude a question. Blocking it stops Claude retrieving your page for that user. Anthropic |
PerplexityBot | Perplexity | Surfaces and links sites in Perplexity's search results. Perplexity says it is not used to crawl for AI foundation models. Perplexity |
Perplexity-User | Perplexity | Visits a page when a user asks Perplexity a question. Perplexity says it generally ignores robots.txt, since a user asked, and is not used for training. Perplexity |
Google-Extended | Not a separate bot: a robots.txt name that controls whether pages Google already crawls may train future Gemini models and ground answers in Gemini Apps. Google says it does not affect inclusion or ranking in Google Search. Google | |
Applebot-Extended | Apple | Does not crawl. It lets you opt out of your content training Apple's foundation models. Pages that block it can still appear in Apple's search results. Apple |
bingbot | Microsoft | Bing's web crawler. Bing Bing says grounding for AI answers "builds on the same foundational infrastructure – the same crawlers" as its search. Bing |
Does blocking an AI crawler stop AI from citing me?
It depends on the bot's job. There are three:
- Training bots (
GPTBot,ClaudeBot, and theGoogle-ExtendedandApplebot-Extendedcontrols). Blocking them is a business choice about training. OpenAI says each of its bot settings is independent, and Google and Apple say their training controls do not affect their search results. - Search bots (
OAI-SearchBot,Claude-SearchBot,PerplexityBot, andbingbotfor Microsoft). Block these and you lose the index those engines cite from. OpenAI says it plainly: opted-out sites are not shown in ChatGPT search answers. - User-triggered fetchers (
ChatGPT-User,Claude-User,Perplexity-User). These fetch your page because someone asked about it right now.
Google has no separate AI Overviews bot. Google says a page only needs to be indexed and eligible for a snippet in Search to appear as a link in AI Overviews or AI Mode. Google Search Central
Sources
- Overview of OpenAI crawlers — OpenAI. Read Sep 24, 2026.
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic. Read Sep 24, 2026.
- Perplexity Crawlers — Perplexity. Read Sep 24, 2026.
- Google's common crawlers — Google Search Central. Read Sep 24, 2026.
- About Applebot — Apple. Read Sep 24, 2026.
- Announcing user-agent change for Bing crawler bingbot — Bing Webmaster Blog. Read Sep 24, 2026.
- Evolving role of the index: From ranking pages to supporting answers — Bing Search Blog. Read Sep 24, 2026.
- AI features and your website — Google Search Central. Read Sep 24, 2026.
Common questions
Is blocking GPTBot the same as blocking ChatGPT?
No. OpenAI says each of its bot settings is independent. Blocking GPTBot keeps your pages out of training; OAI-SearchBot is the one that decides whether you can show in ChatGPT search answers.
Does blocking Google-Extended remove me from AI Overviews?
Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode use pages that are indexed in Search and eligible for a snippet.
Can robots.txt block every AI visit?
Not always. OpenAI and Perplexity both say their user-triggered fetchers may not follow robots.txt, because a person started the request.
Related pages.
See what AI says about your business.
Enter your domain and the free AI visibility check shows where ChatGPT, Gemini, Perplexity and Google's AI answers mention you — and where they don't.