Anthropic · Microsoft · OpenAI · Google · Meta · Apple · The Verge
The biggest names in AI are backing a new way to opt out of scraping
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
The new control will allow publishers to apply their AI training preferences across multiple crawlers.
Key facts
- The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF)
- Currently, sites have to signal their AI training preferences separately to each AI company
- The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models
- Microsoft says it will offer granular controls that will give publishers the ability to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information
Summary
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token.