Cloudflare will block some AI web crawlers by default starting Sept. 15

In short: Cloudflare is changing its default settings so many ad-supported websites will block “mixed-use” AI crawlers unless AI companies separate their bots by purpose.

What happened

Cloudflare said AI companies have until September 15, 2026 to separate the web crawlers they use for search from the crawlers they use for AI training and AI “agents” (tools that try to do tasks for you online).

A web crawler is a bot, meaning an automated program that visits web pages the way a person would, but at huge scale (like a very fast reader that can open millions of pages). Cloudflare says that starting September 15, its default settings will block “mixed-use” crawlers on pages that host ads. Mixed-use means one crawler is doing more than one job, like indexing pages for search while also collecting content for AI training.

Cloudflare said these defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers. Site owners can still change the settings if they want to allow access.

Cloudflare also pointed to its publisher tools, including “Pay Per Crawl,” and said it is evolving this into “Pay Per Use.” The idea is closer to paying royalties when content creates value, not just paying when a bot copies a page.

Why it matters

This could affect what AI products can read from the open web, and it could push AI companies to pay publishers more often. For everyday readers, it may influence which sources show up in AI answers, and whether news and other content stays funded by ads.

Source: TechCrunch AI