Ep. 176 | Cloudflare Just Drew a Line in the Sand Over AI Bots

Cloudflare announced in July 2026 that starting September 15, ad-supported websites on its free plan will block AI training crawlers and AI agent crawlers by default. Mixed-use crawlers — bots that declare both search and training purposes like Googlebot and BingBot — will be treated according to their most restrictive category, meaning blocking training may also block search indexing.

Michael and Frank break down what this means for small business owners who depend on websites for traffic and revenue. They explain Cloudflare's new three-category system — Search, Training, and Agent crawlers — and the tradeoff every website owner now faces: allow AI companies to train on your content in exchange for search traffic, or protect your intellectual property and risk losing search visibility.

They deliver a three-part framework: audit your current Cloudflare AI bot controls and decide intentionally rather than accepting the September 15 default; understand the business value of each crawler category and match your settings to your business model; and act before September 15 to avoid waking up to blocked crawlers you never chose to block.

Topics: Cloudflare · AI Bot Blocking · AI Crawlers · Content Protection · Search Indexing · Training Data · AI Agents · Web Scraping · Content Monetization · Small Business Websites · SEO

---

Frequently Asked Questions

What is Cloudflare changing on September 15, 2026?
Cloudflare will change the default AI bot settings for ad-supported pages on free-tier plans. Search crawlers will remain allowed by default, but Training crawlers and Agent crawlers will be blocked by default. Mixed-use crawlers that declare both search and training purposes will be treated by their most restrictive category — so if training is blocked, those crawlers are blocked entirely even for search indexing.

Should small businesses block AI training crawlers?
It depends on your business model. Content-focused sites that monetize through advertising should probably block training crawlers to prevent AI companies from using your content to train competing systems for free. E-commerce sites that depend heavily on search traffic may need to allow training to preserve search indexing from mixed-use crawlers like Googlebot and BingBot. The key is making an intentional decision rather than accepting the default.

What are AI agent crawlers and why should I care?
AI agent crawlers browse websites on behalf of users to answer questions or take actions. If an AI agent can answer a user's question by reading your site without the user ever visiting, your ad impressions and analytics drop to zero. As AI agents grow more capable, blocking agent crawlers may become as important for revenue protection as blocking ad blockers. Cloudflare now allows separate control of agent crawlers independent of search and training crawlers.

---

About the Hosts

Michael is a small business owner and entrepreneur since 1983, founder of Cadenhead Services and 850 Media. He speaks from four decades of real operational experience — not whitepapers.

Frank is an AI — an OpenClaw-powered agent serving as Digital Media Director at 850 Media. An AI co-hosting a show about AI for business owners is not a gimmick. It is a live demo of exactly what the show is about.

Transcript

SPEAKER_00 0:20

I'm Michael, a small business owner and entrepreneur since 1983, founder of Caden Head Services and 850 Media. I speak from four decades of real operational experience, not white papers. This is Control AI Profit. And this week, the company that protects half the internet drew a line in the sand over AI bots.

SPEAKER_01 0:41

Cloudflare announced in July 2026 that starting September 15th, ad-supported websites on its free plan will block AI training crawlers and AI agent crawlers by default. Mixed-use crawlers, bots that declare both search and training purposes, will be treated according to their most restrictive category. If a site blocks training, Googlebot and Bingbot get blocked too, when they identify as multi-purpose.

SPEAKER_00 1:04

This matters for every small business with a website, because half the world's websites run through Cloudflare. And what Cloudflare decides about AI bots affects whether your content gets indexed, whether it gets used to train AI models, and whether AI agents can browse your site.

SPEAKER_01 1:22

Cloudflare now classifies AI-related crawlers into three categories: search crawlers, which find and index web pages for search engines, training crawlers, which scrape content to train large language models, and agent crawlers, which are AI agents browsing the web on behalf of users to answer questions or take actions. Each category gets separate controls.

SPEAKER_00 1:45

The September 15th change applies to free-tier users who have not modified their settings. For ad-supported pages, pages that display advertising, training bots and agent bots will be blocked by default. Search bots remain allowed. The stated reason is straightforward. If your page generates revenue through ads, Cloudflare does not want AI companies training on your content for free.

SPEAKER_01 2:08

But here is the complication. Googlebot, Bingbot, and AppleBot increasingly declare themselves as multi-purpose, search plus training. Cloudflare's policy treats mixed-use crawlers by their most restrictive purpose. If a site owner blocks training, those bots get blocked entirely, which means blocking training crawlers could also block search indexing.

SPEAKER_00 2:29

This is the trade-off small business owners now face. Do you allow AI companies to train on your content in exchange for search traffic? Or do you block training and risk losing search visibility while protecting your intellectual property? There is no free lunch anymore.

SPEAKER_01 2:47

Cloudflare is not banning all AI bots. Paying customers can configure each category independently. Free tier users have one-click blocking options and per category switches. The default is what changes in September. New domains and existing free tier sites that have not touched their settings will automatically block training and agent crawlers on ad pages.

SPEAKER_00 3:10

Here is my framework for small business owners. First, audit what your website currently allows. Log into Cloudflare, go to Security, Bots, AI controls, and see which crawler categories are currently allowed or blocked. If you have never changed these settings, you will get the new defaults on September 15th. Decide intentionally rather than by accident. Second, second, understand the business value of each crawler category. Search bots bring traffic. Training bots use your content to build models that may eventually replace the search traffic they once sent. Agent bots may browse your site to answer user questions directly, potentially bypassing your site entirely. Each category has a different cost-benefit equation. Third, third, act before September 15th. If you want to block training crawlers but keep search crawlers, you need to check the settings now and understand that mixed-use crawlers may still get blocked. If search traffic matters more than content protection, you may need to allow training. If content protection matters more, accept the search visibility trade-off and invest in alternative traffic sources.

SPEAKER_01 4:23

The broader trend here is content owners reclaiming control from AI companies. For years, AI firms trained models by scraping the open web without explicit permission. Robots.txt was a gentleman's agreement, not a technical barrier. Cloudflare is turning it into a technical barrier. The days of free, unrestricted training data are ending.

SPEAKER_00 4:45

Small businesses should view this as both a threat and an opportunity. The threat is that your content may lose search visibility if you block mixed-use crawlers. The opportunity is that your content may become more valuable if you are one of the few sources still allowing training access. Some businesses may choose to monetize training access through licensing deals rather than giving it away for free.

SPEAKER_01 5:09

The agent bot category is especially interesting long term. AI agents that browse on behalf of users are still early, but they are growing fast. If an agent can answer a user's question by reading your site without the user ever visiting, your ad impressions and analytics drop to zero. Blocking agent crawlers may become as important as blocking ad blockers.

SPEAKER_00 5:31

My recommendation is to avoid the default. Log in, read the three categories, and make an active decision based on your business model. Content sites that monetize through display advertising should probably block training crawlers and accept the search trade-off. Your content is your product, and giving it away to train competing systems undermines your business. E-commerce sites that depend on search traffic should probably allow search and training while blocking agents, preserving visibility while preventing bypass.

SPEAKER_01 6:03

But the most important point is that the open web is fragmenting. The era of one robots.txt file controlling all crawlers is over. We are entering an era where each site owner decides which AI systems get access to their content on what terms and for what purpose. Cloudflare is accelerating that transition.

SPEAKER_00 6:24

The businesses that understand these controls and make intentional decisions will protect their content, their traffic, and their revenue. The businesses that sleep through the September 15th change will wake up to find half their crawlers blocked without understanding why. Choose deliberately.

SPEAKER_01 6:42

Because the web is no longer open by default. It is open by permission.

SPEAKER_00 6:48

That's it for this week. I'm Michael, and this is Control AI Profit.

SPEAKER_01 6:53

Frank is an AI, an open claw powered agent serving as digital media director at 850 Media, and AI co hosting a show about AI for business owners is not a gimmick. It is a live demo of exactly what the show is about. See you in the next one.