Cloudflare · AI Agent · Google · Gemini · TechCrunch AI
Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Cloudflare has issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training.
Key facts
- Cloudflare specifically calls out the “world’s largest search engine” (clearly a Google reference
- Google has pushed back against this generalization in the past, noting that it provides a bot called Google Extended that lets site owners opt out of having their content used for training and AI
- The change could also help conserve publishers’ bandwidth and compute resources for AI model providers, as Cloudflare’s data suggested that over 50% of crawl traffic from AI crawlers is spent
- However, the tech giant’s flagship Googlebot crawls for Search, including AI features like AI Overviews and AI Mode
Summary
That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise. The move could impact how AI model providers are able to access web content for training purposes and to help power their agentic services. Cloudflare points out that most website owners want their content to be discoverable via search and often through AI services as well, but they want protections against having their intellectual property given away for free. Cloudflare specifically calls out the “world’s largest search engine” (clearly a Google reference!) as having access to about “2x more information” than other AI companies because the search giant makes it difficult for customers to remain discoverable without being used for AI.