Cloudflare's September 15 Deadline Is Coming — And It Could Accidentally Block Googlebot on Your Site

Here's a scenario nobody should want to live through: you wake up after September 15, check Search Console, and your ad-supported pages are getting zero impressions. Not a rankings drop — a full crawl block. No indexing at all. And the culprit? A Cloudflare setting you enabled 18 months ago and never thought about again.

That's not a hypothetical. It's exactly what could happen to sites running on Cloudflare that turned on the old "Block AI Bots" toggle — and haven't reviewed it since. On July 1, 2026, Cloudflare announced a significant overhaul of how it handles AI crawler traffic. The new system is genuinely better and more nuanced than what came before. But it has one sharp edge that can cut you if you're not paying attention: Googlebot is now classified as a mixed-use crawler, and mixed-use crawlers get hit with the strictest applicable rule.

Let's walk through exactly what changed, why Googlebot is in the crossfire, and what you need to do before the September 15 deadline.

Why Cloudflare Changed the System in the First Place

The original "Block AI Bots" toggle made sense in 2024 when there were maybe a dozen AI crawlers and they all basically did the same thing: scrape your content to train models. The world has gotten considerably more complicated since then.

Cloudflare CEO Matthew Prince recently confirmed something that caught a lot of people off guard: bot traffic has now surpassed human traffic on the internet. He didn't expect that milestone until late 2027. It happened ahead of schedule, driven almost entirely by AI. AI training crawlers went from roughly 20% of Cloudflare's network traffic in spring 2025 to becoming the majority today. Daily AI agent requests grew by over 1,700% in a single year.

The old toggle couldn't handle that complexity. You'd block GPTBot (which trains models and returns zero traffic to you) and also accidentally block OAI-SearchBot (which powers ChatGPT search and actually sends referral traffic). Same company, completely different behavior, one switch for both. That's why Cloudflare broke it into three distinct categories.

The Three Bot Categories You Need to Know

Search bots crawl and index your content so AI-powered search experiences can answer questions with it later. When someone gets a citation in Perplexity or a source link in ChatGPT search, a Search bot is what put you there. These bots return referral traffic. Cloudflare keeps them allowed by default and won't change that on September 15.

Agent bots act in real-time on behalf of a specific person. When someone uses ChatGPT to "check this website for me," that's an Agent bot visiting your site. No index being built, no future referral traffic — just one person getting your content without ever visiting. On ad-supported pages, that's pure revenue loss with no upside.

Training bots scrape your content to train or fine-tune AI models. Your writing goes into the model's weights permanently. No attribution, no traffic, no compensation (unless you've worked out a licensing deal). This is the category publishers have been fighting about for two years.

Starting September 15, Agent and Training bots will be blocked by default on ad-supported pages for new Cloudflare domains and existing free-tier customers who haven't adjusted their settings. That part is fine. The problem is what happens to crawlers that do multiple things.

The Googlebot Problem Nobody Is Talking About Clearly

Googlebot isn't just a search crawler. Google uses the same bot — the one indexing your pages for traditional search — to also collect content for Gemini, AI Overviews, and AI Mode. One bot, two jobs: Search indexing and AI Training.

Cloudflare's new rule for multi-purpose crawlers: apply whichever restriction is strictest. So if you have Training bots blocked on ad pages, and Googlebot does Training, then Googlebot is blocked on your ad pages. Full stop. Your ad-monetized content won't be crawled. Those pages drop out of the index.

Cloudflare explicitly named Googlebot, Applebot, and Bingbot as examples of crawlers affected by this rule. And here's the thing that makes it sneaky: a Cloudflare-level block is harder to work around than a robots.txt directive. Google's crawler has been known to crawl pages that robots.txt tells it to skip. A network-level block through Cloudflare? That's a wall, not a suggestion.

If you turned on "Block AI Bots" at any point in the past and haven't revisited that setting, check it now. You may already be configured to block Googlebot on your ad pages after September 15 rolls out the new multi-purpose crawler rules.

How to Fix It: Your 5-Step Cloudflare Configuration

Good news: the controls are free on every Cloudflare plan, including the free tier. This takes about 15 minutes if you know where to look.

Step 1 — Find the AI Crawl Control panel

Log into your Cloudflare dashboard. Select your domain. Navigate to Security → Bots → AI Crawl Control. You'll see two tabs: Robots.txt and Crawlers. Start with Crawlers.

Step 2 — Check your legacy settings

If you ever enabled the old "Block AI Bots" toggle, look for it here. If it's still on, you're in the group that gets automatically migrated to the new strictest-rule logic on September 15. You need to explicitly set your preferences before that date or opt out of the automatic migration.

Step 3 — Set your three category controls

For most ad-supported content sites, the right configuration is:

• Search bots: Do not block — these drive referral traffic and AI search visibility.
• Agent bots: Block on pages that display ads — they return no value on monetized pages.
• Training bots: Block on pages that display ads — protects revenue without killing your full-site indexing.

If you're a SaaS site, newsletter, or subscription product where every page matters and ad impressions aren't the revenue driver, consider blocking Training bots on all pages — not just ad pages.

Step 4 — Handle Google separately via robots.txt

This is the clean way to deal with Googlebot's dual-purpose nature. Google provides a separate crawler called Google-Extended specifically for AI training opt-outs. By adding Disallow: / for Google-Extended in your robots.txt, you tell Google not to use your content to train its AI models — without blocking Googlebot from indexing you for search.

One important caveat worth knowing: disallowing Google-Extended doesn't remove you from AI Overviews. Those pull from Google's regular Search index, not a separate training crawl. There's currently no way to appear in AI Overviews while fully opting out of Google's AI training use. That's a gap Google hasn't addressed yet. Disallowing Google-Extended is still worth doing — it's just not a complete solution to the AI training question.

Step 5 — Check for WAF rule conflicts

Cloudflare's Web Application Firewall rules run before robots.txt is even read. If you have any WAF custom rules that challenge or block user agents matching AI crawlers, they can override your AI Crawl Control settings. Go to Security → WAF → Custom Rules and check for anything that might be catching legitimate Search bots like OAI-SearchBot or PerplexityBot. If you find them, create explicit skip rules for their verified user agents.

What About the "Block All AI Bots" Crowd?

If you're in the camp of "I don't want any AI touching my content, period," I get it. But blocking all AI bots including Search bots has a cost that's worth being clear-eyed about: you're removing your content from the AI search ecosystem entirely.

Perplexity, ChatGPT search, and AI-enhanced Google results all depend on Search bots finding your content. If you block them, you won't show up in those results. For some sites — especially those with content that competes directly with what AI systems generate — that trade-off might be worth it. But for most publishers and marketers, cutting off all AI search visibility to block Training bots is overkill.

The smarter play is the nuanced one: block Training and Agent bots on ad pages (protecting your revenue), keep Search bots allowed (keeping your AI search visibility), and use Google-Extended in robots.txt to signal your AI training preferences to Google directly. You're not giving up the whole game — you're just protecting the part that actually funds your site.

Cloudflare's Pay Per Use: Worth Signing Up For?

There's a new piece to this story that's been buried under the September 15 deadline news. Cloudflare launched what they're calling the Monetization Gateway — an evolution of their earlier Pay Per Crawl experiment. The new model is called Pay Per Use, and it charges AI companies based on when your content actually appears in a chatbot answer, not just when it was crawled.

In theory: if GPT-4o uses your article to answer someone's question, you get paid for that usage. Payments settle in stablecoins via Cloudflare's x402 protocol. Early partnerships include a handful of smaller AI platforms — the big names (OpenAI, Google, Anthropic) aren't on board yet.

Should you sign up for the waitlist? Yes, with realistic expectations. The infrastructure is being built, but it's not a meaningful revenue stream in 2026 for most sites. The bigger deal right now is using the bot controls to protect your ad revenue. Treat Pay Per Use as a long-term upside — sign up for the waitlist and forget about it until the landscape matures.

Bottom Line

You've got about two months before September 15 makes this urgent — but honestly, there's no reason to wait. Log into your Cloudflare dashboard today and spend 15 minutes reviewing your AI Crawl Control settings. The check is free, the fix is free, and the downside of ignoring it is real: pages that stop getting indexed, rankings that quietly evaporate, all from a setting you turned on once and forgot about.

The configuration for most sites is straightforward: Search bots allowed everywhere, Agent and Training bots blocked on ad pages, Google-Extended disallowed in robots.txt. That's it. You're protecting your ad revenue, keeping Googlebot happy, and maintaining your presence in AI search — all at once.

Do it now, not because September 15 is a hard cliff, but because it takes 15 minutes and the alternative is finding out what happened after it's already cost you traffic.