Is Cloudflare blocking AI from your website?
Cloudflare says it helps manage and protect traffic for 20% of the web, and it has changed how it treats AI bots twice: in July 2025 and again on 15 September 2026. Depending on when your site joined and what someone clicked, it may be turning away the AI assistants your customers use. Here is what each setting does, how to check yours, and how to keep AI search while saying no to training.
Possibly. Since July 2025, Cloudflare asks every new domain whether to allow AI crawlers, and since 15 September 2026 ad-supported sites are offered settings that block AI agents on pages with ads. Check Security Settings for the Search, Training and Agent controls. Keep Search and Agent on Allow, and use Disallow AI Training, not Block, to refuse training.
What are the key facts about Cloudflare and AI crawlers?
| Fact | Detail | Source |
|---|---|---|
| Blocking by default | On 1 July 2025 Cloudflare said it was the first infrastructure provider to block AI crawlers "without permission or compensation, by default." Every new domain is asked at sign-up whether to allow AI crawlers | Cloudflare |
| The one-click block came first | Added in September 2024; "More than one million customers" had chosen it by July 2025 | Cloudflare |
| Three controls since 15 Sep 2026 | The single "Block AI Bots" switch was replaced by separate Search, Training and Agent controls, on all plans | Cloudflare |
| Preset for ad-supported sites | Search allowed, AI training disallowed, AI agents blocked on pages that show ads | Cloudflare |
| "Block" now reaches search | Block on Training now also stops Googlebot, Bingbot and Applebot, "search included" | Cloudflare |
| The old switch was migrated | Sites that had turned on legacy "Block AI" now have Agent set to "Block on pages with ads" | Cloudflare |
| Few sites block search | "less than 1%" of Cloudflare sites block search bots; 17% use some way to block training | Cloudflare (Sep 2026) |
| Mixed-use crawlers | One crawler doing both search and training: 36.6% of verified crawler traffic on Cloudflare's network | Cloudflare (Sep 2026) |
| Bot Fight Mode | Free, but you "cannot bypass or skip" it with WAF custom rules | Cloudflare |
| Pay per crawl | Closed beta; a crawler either pays or gets "402 Payment Required" | Cloudflare |
| Separate bots, separate jobs | OpenAI, Anthropic and Perplexity each run a search crawler and a user-triggered fetcher apart from any training crawler | OpenAI, Anthropic, Perplexity |
Is Cloudflare blocking AI crawlers on my website?
It might be, and you can't tell from the outside. It depends on when your domain joined Cloudflare and which settings someone picked. Cloudflare is a CDN, a network that sits between visitors and your web server, so it can turn a bot away before your site ever sees the request.
Here is the short history, from Cloudflare's own announcements.
- September 2024. Cloudflare added an option to block AI crawlers "in a single click." By July 2025 it said more than one million customers had chosen it. Cloudflare
- 1 July 2025. Cloudflare began asking every new domain at sign-up whether to allow AI crawlers, so each one "starts with the default of control." Cloudflare
- 15 September 2026. Cloudflare replaced its "Block AI Bots" switch with three controls: one for search, one for AI training and one for AI agents. New domains are offered a preset. Sites that carry ads get search on, training disallowed, and agents blocked on pages with ads. All other sites get all three allowed. Cloudflare
The part that catches people is the old switch. If anyone on your team ever turned on "Block AI Bots," Cloudflare migrated that to Training set to Disallow AI Training and Agent set to Block on pages with ads. Cloudflare On any page Cloudflare detects is showing ads, that turns away ChatGPT-User, Claude-User and Perplexity-User. Those are the fetchers an assistant sends when a person asks it about you right now.
What is the difference between AI training, search and agent bots?
Training bots collect pages to build future AI models. Search bots build the index an AI engine looks things up in. Agent bots open one page because a person asked a question right now. Cloudflare sorts AI bots into the same three behaviors, and says "A single bot can have more than one behavior." Cloudflare
| Company | Training | Search | Agent (user-triggered) |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed. PerplexityBot "is not used to crawl content for AI foundation models" | PerplexityBot | Perplexity-User |
Sources: OpenAI, Anthropic, Perplexity.
Blocking the wrong one costs you different things:
- Block a search bot and you drop out of that engine's answers. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Anthropic says disabling Claude-SearchBot "may reduce your site's visibility." OpenAI Anthropic
- Block an agent and live questions about you go unanswered from your site. Anthropic says disabling Claude-User "prevents our system from retrieving your content in response to a user query." Anthropic
- Block a training bot and nothing changes in search. OpenAI says "Each setting is independent of the others," so you can allow OAI-SearchBot and disallow GPTBot. OpenAI
Google, Microsoft and Apple work differently. Cloudflare calls Googlebot, Bingbot and Applebot "mixed-use crawlers," meaning one crawler collects pages for both search and training. Cloudflare Google gives you a separate training opt-out, the Google-Extended robots.txt token, and says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Google
What do Cloudflare's Search, Training and Agent settings do?
Each of the three controls can be set to one of these options. Cloudflare describes them like this. Cloudflare
| Option | What it does | Offered for |
|---|---|---|
| Allow | All crawlers allowed, unless another setting or a WAF rule blocks them | Search, Training, Agent |
| Disallow AI Training | Publishes a no-training rule in your robots.txt. Googlebot, Bingbot and Applebot stay allowed for search. Every other training crawler is blocked, including GPTBot and ClaudeBot | Training only |
| Block on pages with ads | Blocks only on pages Cloudflare detects are serving an ad | Search, Training, Agent |
| Block | Blocks the whole domain | Search, Training, Agent |
The word Block changed meaning on 15 September 2026. Before then, Block on Training skipped mixed-use crawlers. Now it doesn't. Cloudflare says choosing Block "will stop Applebot, Bingbot, and Googlebot from reaching your site — search included." To refuse training and keep search, it says to use Disallow AI Training instead. Cloudflare
One gap to know about. Until Microsoft supports a robots.txt no-training rule, which Cloudflare says is targeted for early 2027, Disallow AI Training does not pass a no-training preference to Bing. Cloudflare points to Bing's NOARCHIVE meta tag for now. Cloudflare
Per-crawler switches live in AI Crawl Control. This dashboard is "Available on all plans" and lists each AI crawler with an Allow or Block action. Cloudflare Blocking a crawler there creates or updates a WAF custom rule on your domain. On paid plans you can choose whether blocked crawlers get a 403 Forbidden or a 402 Payment Required response. Cloudflare
Pay per crawl lets a site charge AI crawlers per page. It is in closed beta. A crawler either pays and gets the page, or receives a 402 response with the price. Cloudflare
Can Bot Fight Mode block AI search bots?
Yes, it can challenge them, and it is the hardest Cloudflare setting to make exceptions to.
Bot Fight Mode is free. When it is on, Cloudflare "Issues computationally expensive challenges" to traffic matching known bot patterns. Cloudflare warns it "may challenge API or mobile app traffic," and says "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules." Cloudflare
Super Bot Fight Mode comes with Pro, Business and Enterprise plans. It lets you choose an action for each group of bots, including Verified bots, and you can exclude traffic with a WAF custom rule using the Skip action. Cloudflare
A verified bot is one Cloudflare has confirmed is honest about who it is, using a cryptographic signature, a published IP list, or reverse DNS. Cloudflare says verified bots have "historically" been excluded in its default bot settings. Cloudflare
A challenge is a dead end for well-behaved AI bots. Anthropic says its bots "will not attempt to bypass CAPTCHAs." Anthropic
What does Cloudflare's managed robots.txt tell AI crawlers?
It asks known training crawlers to stay out and leaves search bots alone. A robots.txt file sits at yourdomain.com/robots.txt and tells crawlers which pages they may read.
In Cloudflare's own example, the managed file adds Disallow: / for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. It also adds a content signal line: search=yes, ai-train=no. It does not name OAI-SearchBot, Claude-SearchBot, PerplexityBot or the user-triggered fetchers. Cloudflare
If your site already has a robots.txt, Cloudflare puts its rules in front of yours, between BEGIN Cloudflare Managed content and END Cloudflare Managed Content lines. Your own lines below still apply. Cloudflare
Cloudflare also says "robots.txt compliance is voluntary": the file states a wish but stops no one. Cloudflare And managed robots.txt is being replaced by Bot Preference Sync, with existing users migrated. Cloudflare
Disallow written years ago for every bot can do more harm than anything Cloudflare adds. For which AI bots to name and why, see Which AI crawlers to allow.How do I check if my site is blocking AI search engines?
Look at what Cloudflare is doing to the real bots, in this order. It takes about 15 minutes if you have the login.
- Confirm you're on Cloudflare. Open your site, then your browser's developer tools, and look at the response headers. Cloudflare returns a
Cf-Rayheader to visitors. Cloudflare If you don't see it, ask whoever manages your domain which CDN or firewall sits in front of it. - Read your three AI controls. In the dashboard, go to Security Settings and filter by Bot traffic. Write down Search, Training and Agent, and whether Bot Fight Mode or Super Bot Fight Mode is on. Cloudflare Cloudflare
- Open AI Crawl Control, Crawlers tab. Find
OAI-SearchBot,ChatGPT-User,Claude-SearchBot,Claude-User,PerplexityBotandPerplexity-User. Check the Action column. The Requests column counts allowed and unsuccessful requests, and Cloudflare notes unsuccessful ones "may come from any rule or response error." Cloudflare - Mind the free-plan limits. On the free plan, AI Crawl Control spots crawlers by their user-agent name, and the Metrics tab only shows the past 24 hours. Cloudflare
- Read your robots.txt at
yourdomain.com/robots.txt, as described above. - Check Security Events for challenges or blocks on AI bots.
- Ask the engines. Our Prompt Simulator runs one question across 12 AI engines and shows which sources each one cites. Our free AI Access scan checks crawler access along with 30+ other checks per page.
OAI-SearchBot into a test tool doesn't make your request come from OpenAI. Cloudflare recognises verified bots by signature, published IP list or reverse DNS. Cloudflare So your test is judged as a stranger using a famous name, and the result can differ from what the real bot gets. This is our advice: trust the Crawlers tab and your server logs over a spoofed request.What should I change so AI search can read my site but training stays blocked?
Allow search and agents, and use Disallow AI Training if you want training off. This table is our advice, built on Cloudflare's own definitions above.
| Setting | Set it to | Why |
|---|---|---|
| Search | Allow | Blocking it removes you from search, including Google and Bing |
| Training | Disallow AI Training to refuse training, or Allow if you're happy for models to learn from you | Disallow AI Training keeps Googlebot, Bingbot and Applebot crawling for search. Block does not |
| Agent | Allow | These are the fetchers sent when someone asks an assistant about you |
| Bot Fight Mode | Off, if Security Events show it challenging AI bots | You can't add exceptions to it |
| Super Bot Fight Mode | Verified bots: Allow | Keeps honest, identified bots out of the challenge |
| AI Crawl Control | Allow for OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User | A per-crawler block overrides everything else |
| Your own WAF rules | Remove broad rules that block "bot" or unknown user agents | Old custom rules often outlive the reason they were added |
If your site earns money from ads, blocking agents on ad pages is a fair business choice. Cloudflare's reasoning is that agents "fetch the page with nobody there to see the ads." Cloudflare Just know that an assistant that can't open your page will answer from other sources.
If you want Bing out of training too, add the NOARCHIVE tag Cloudflare points to, until Microsoft's robots.txt support arrives. Cloudflare
Give changes a day. OpenAI says a robots.txt change "can take ~24 hours" to reach its search systems, and Perplexity says "up to 24 hours." OpenAI Perplexity Then recheck every engine's bots, not just ChatGPT's. See How to get cited by every AI engine.
Can other CDNs, hosts and firewalls block AI bots too?
Yes. Any layer between a bot and your page can turn it away, including other CDNs, hosting firewalls and security plugins. Two documented examples:
- Vercel. Its AI Bots Managed Ruleset is "available on all plans" and inactive by default. Its Deny action "blocks all traffic identified as coming from AI bots," so as documented it is one switch, with no separate search and training settings. Vercel's Bot Protection rule, in challenge mode, serves "a JavaScript challenge to traffic that is unlikely to be a browser." Vercel
- AWS WAF and Cloudflare WAF. Perplexity publishes step-by-step allow rules for both, and recommends matching the bot's user-agent and its published IP addresses together. Perplexity
Sources
- Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model — Cloudflare. Read Oct 7, 2026.
- Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules — Cloudflare. Read Oct 7, 2026.
- Cloudflare Helps End the Search-or-AI-Training Tradeoff — Cloudflare. Read Oct 7, 2026.
- Have it both ways: stay discoverable in search while disallowing AI training — Cloudflare Blog. Read Oct 7, 2026.
- Block AI Bots — Cloudflare Docs. Read Oct 7, 2026.
- Bots — Cloudflare Docs. Read Oct 7, 2026.
- Verified bots — Cloudflare Docs. Read Oct 7, 2026.
- AI Crawl Control — Cloudflare Docs. Read Oct 7, 2026.
- Manage AI crawlers — Cloudflare Docs. Read Oct 7, 2026.
- Get started (AI Crawl Control) — Cloudflare Docs. Read Oct 7, 2026.
- robots.txt setting — Cloudflare Docs. Read Oct 7, 2026.
- Bot Fight Mode — Cloudflare Docs. Read Oct 7, 2026.
- Super Bot Fight Mode — Cloudflare Docs. Read Oct 7, 2026.
- What is Pay Per Crawl? — Cloudflare Docs. Read Oct 7, 2026.
- Cloudflare HTTP headers — Cloudflare Docs. Read Oct 7, 2026.
- Overview of OpenAI Crawlers — OpenAI. Read Oct 7, 2026.
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic. Read Oct 7, 2026.
- Perplexity Crawlers — Perplexity. Read Oct 7, 2026.
- Google's common crawlers — Google for Developers. Read Oct 7, 2026.
- WAF Managed Rulesets — Vercel. Read Oct 7, 2026.
Common questions
Does Cloudflare block ChatGPT by default?
Not all of it. Since 15 September 2026, new domains that carry ads are offered a preset that allows search (OAI-SearchBot), disallows training (GPTBot) and blocks agents such as ChatGPT-User on pages with ads. Other new sites are offered all three allowed. Check Security Settings to see yours. Cloudflare
Will blocking GPTBot remove my site from ChatGPT search?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says "Each setting is independent of the others." OpenAI
What's the difference between Block and Disallow AI Training on Cloudflare?
Disallow AI Training publishes a no-training rule in robots.txt, keeps Googlebot, Bingbot and Applebot crawling for search, and blocks other training crawlers such as GPTBot and ClaudeBot. Since 15 September 2026, Block on Training also stops Googlebot, Bingbot and Applebot, "search included." Cloudflare
Does Cloudflare's managed robots.txt block AI search engines?
No. In Cloudflare's example it disallows training crawlers such as GPTBot, ClaudeBot and Google-Extended, and sets search=yes, ai-train=no. It doesn't name OAI-SearchBot, Claude-SearchBot or PerplexityBot. Cloudflare is replacing it with Bot Preference Sync. Cloudflare Cloudflare
Should I turn off Bot Fight Mode?
Only if it is challenging bots you want. Check Security Events for requests labelled "Bot Fight Mode." Cloudflare says you can't make exceptions to it with WAF custom rules. Super Bot Fight Mode, on paid plans, lets you allow verified bots and add Skip rules. Cloudflare Cloudflare
Related pages.
See whether AI engines can read your site.
Enter your domain. The free AI visibility check shows where ChatGPT, Claude, Gemini, Perplexity and Google's AI answers mention you, and where they don't.