Cloudflare manages AI bots by behavior, not by name: it distinguishes Search, Agent, and Training. For each category you choose whether to block it everywhere, only on pages with ads, or not at all. Starting September 15, 2026, new domains get different defaults and the old toggle retires.
If you’ve ever read a guide along the lines of “block Bytespider, allow GPTBot,” it’s out of date. What Cloudflare offers today is control by behavior category, not by individual bot name.
What changed, and why there’s a deadline
Since July 1, 2026, every customer — including those on the free plan — can configure AI bot behavior separately for three categories. Each one comes with three options: block on all pages, block only on pages with ads, or don’t block.
Right now, per Cloudflare, all three categories are allowed by default. That changes on September 15, 2026 for newly created domains: those are set to get Training and Agent blocked on pages with ads, and Search allowed. The same date retires the old Block AI bots toggle — the one guides have been recommending for years.
Three behaviors instead of a list of names
| Aspekt | What it is | What it means for you |
|---|---|---|
| Search | Behavior that collects or indexes content so it can answer questions about it later | Per Cloudflare, the behavior that most naturally sends visitors back |
| Agent | Automated activity acting in real time on behalf of a specific person | This covers assistants that handle tasks for a user |
| Training | Crawling that takes content to train or fine-tune a model | The content is permanently absorbed into the model; don't expect a visit |
The difference between them isn’t academic. As its reason for leaving Search allowed even in the new defaults, Cloudflare says it’s the behavior that most naturally sends visitors back, and that allowing it is in most site owners’ interest.
Why blocking Training may not separate cleanly from Search
Here’s the most important sentence in this article, and it comes straight from Cloudflare’s announcement: blocking the Training category also hits bots that combine Search and Training.
The old toggle behaved the opposite way: per the documentation, it left mixed bots alone. So the old and the new option don’t do the same thing — you can’t just carry your settings over and assume the same result.
How to set it up
-
Open AI Crawl Control for the specific site
-
Use the Crawlers tab to see who's visiting
-
Decide each category separately
-
Verify the mixed bots
Cloudflare’s interface keeps changing, so anchor on the name AI Crawl Control rather than a specific menu path. Alongside the Crawlers tab you’ll also find an overview and metrics.
How to decide on each category
There’s no universal answer, but there is a decision guide. And if you’re wrestling with the bigger question — whether you want to be in AI answers at all — there’s a decision framework for opting out.
- Mainly want to limit how your content gets used to train models? Deal with the Training category — and factor in what’s written above about mixed bots.
- Care about AI tools being able to cite you? Then Search is the category you don’t want to close off first.
- Do you make your money from ads? The “block only on pages with ads” option exists for exactly this case.
- No idea what the AI channel is bringing you? That’s the most common situation — and the best first step isn’t configuring anything, it’s measuring.
How this relates to robots.txt
The two don’t conflict at all; they’re two layers stacked on top of each other:
- Robots.txt is a request. A well-behaved bot respects it, the rest ignore it. Robots.txt for AI crawlers walks through it in detail.
- The CDN is enforcement. The bot never reaches the content, whether or not it cares about robots.txt.
Having both makes sense, and for most site owners a single sentence covers it: robots.txt is the signal, Cloudflare is the enforcement.
Common mistakes
Blocking training and expecting citations to survive
Mixed bots fall under the training block. The two can’t always be separated.
Leaving the defaults without knowing what they are
Today everything is allowed; from September 15, new domains get something else. Go look at what you have.
Blocking by bot name
Anyone can put anything in a user-agent header, so name-based blocking is leaky. Cloudflare’s documentation distinguishes verified bots from unverified ones.
Relying on a single layer
Robots.txt on its own enforces nothing; a CDN block on its own gives well-behaved bots no signal.
Where this article stops
It doesn’t cover other CDNs. This is a Cloudflare guide; elsewhere the setup looks different.
It doesn’t map bot names to categories. Cloudflare’s documentation doesn’t assign them that way, so we won’t do it here either.
The takeaway
The change is mostly conceptual: instead of guessing which bot is the bad one, you’re making decisions about three behaviors. The question “do I want something to be able to fetch my content and answer a question with it?” is easier to answer than “should I block CCBot?”
Three practical things to take with you. Go look at what your site is set to today. With the Training category, count on it hitting mixed bots too. And if you have no idea what the AI channel is bringing you, the first step isn’t configuring — it’s measuring.
If what you need first is to find out whether anyone is reaching you from AI tools at all, start with measurement: an AI SEO audit from Sniper Design covers both measurement and visibility, and gives you something concrete to decide from.
Sources: Cloudflare’s “Your site, your rules” announcement, the changelog from July 1, 2026, and the AI Crawl Control documentation. Accurate as of July 19, 2026 — settings and dates can change; we’ll keep this article updated.
