Cloudflare · Google · Engadget
Cloudflare has unveiled aims to automatically block mixed-tap web crawlers that index websites for search engines and act as AI
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
"Now that most traffic on the Internet is non-human, Engadget must go further and act faster so that a sustainable ecosystem can emerge," Matthew Prince, Cloudflare's CEO and co-founder shared in a statement.
Key facts
- Starting September 15, 2026, new customers and new websites from existing Cloudflare subscribers will default "to allow for search but block training and agent use for pages with ads
- As part of these changes, Cloudflare is also releasing a new version of the Pay Per Crawl feature it introduced in 2025 that allowed websites to block AI web crawlers by default unless companies paid
- AI and You.com, but Cloudflare likely hopes other AI companies will join as its customers opt
- The hosting platform wants sites to have more control over how AI companies use their content
Summary
The hosting platform wants sites to have more control over how AI companies use their content. Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. Web traffic used to indicate that people were viewing a website's ads or paying for its subscriptions, but the popularity of AI models that can visit sites on a user's behalf to pull up-to-date information has upended that system. Starting September 15, 2026, new customers and new websites from existing Cloudflare subscribers will default "to allow for search but block training and agent use for pages with ads.