Cloudflare gives AI crawlers a September deadline: pay up or get blocked
From 15 September, Cloudflare will block crawlers that harvest content for AI training from any page carrying ads, unless the owner opts in, and pay publishers when their work shapes an AI answer. It is the boldest bid yet to make AI contribute to the open web.
Timeline and Scope
- July 2, 2026: Cloudflare announces the change.
- September 15: New Cloudflare sites will block AI training and AI agents from ad-supported pages by default.
Key Points
-
Blocking AI Crawlers: This includes crawlers that harvest content for AI training and mixed-use bots that blend search, training, and agent tasks.
-
Opt-In Requirement: Sites can allow the bots back in through their dashboard, but the default position has flipped—content that earns money is now off-limits to AI unless its owner opts in.
-
Motivation: Cloudflare argues that automated bots drive over 50% of all web traffic, and most internet traffic is now non-human. They aim to create a sustainable ecosystem.
-
Targeted at Large Search Engines: Cloudflare singles out the "world’s largest search engine" (a clear reference to Google) for blending indexing with AI training, giving Google significant data advantages over rivals.
-
New Model: Pay Per Use
- Blocking and Payment: Cloudflare is transforming its "Pay Per Crawl" system into "Pay Per Use," paying publishers when their content contributes to an AI answer, not just for the crawl.
- Partnerships: Early partners include Ceramic.ai and You.com.
-
Transparency and Control: Cloudflare introduces a dashboard for publishers to track which bots take their work and how little traffic those firms send back, termed Answer Engine Optimisation (the AI era’s successor to SEO).
Industry Implications
Cloudflare’s move comes as the market shifts towards GEO (Answer Engine Optimisation), with startups offering tools to keep brands visible inside chatbots. Cloudflare aims to control this plumbing.