Cloudflare · AI Agent · Amazon · Meta · TechCrunch AI
The next hurdle for AI agents: getting websites to let them in
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Personal AI agents, like Meta’s Muse, Instinct, ChatGPT’s Dots, and others, are kicking off a new wave of consumer AI that involves more than responding to queries.
Key facts
- This is attributed to a change to crawler defaults on September 15, which allows site owners to block AI training while allowing other types of bots
- Adidas, Zillow, and Pizza Hut either didn’t respond or declined to provide a comment by the time of publication
- It pointed them instead to its free public data hub, Cloudflare Radar
- The company said it wants to be where its customers are, and that now includes being available through AI agent experiences
Summary
But consumers adopting these agents are often running into issues when the website on the other end of their request blocks the AI from completing the job. Most recently, Amazon began blocking Meta’s Muse AI agent from its retail site, meaning that the agent could no longer browse or make purchases from its product catalog. However, the problem is not limited to Amazon or Muse. What’s not clear to the end user is whether or not the block is accidental on the website owner’s part, which leaves them frustrated with both the service provider or retailer and the agent trying to facilitate the transaction.