Anthropic · Demis Hassabis · Pentagon · OpenAI · Google · China · Fortune Technology
It’s ‘more likely than not’ humanity drops control
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Jacob Coxon, a former OpenAI and Anthropic researcher, told the New York City Council on Monday humanity is more likely than not to lose control of advanced AI, doubling down on the warning he issued when he quit last month.
Key facts
- Monday’s hearing was also the first time the City Council held a Committee of the Whole hearing (a hearing of all 51 council members) since 2022, called to weigh a package of AI bills from Speaker
- Often when AI development is discussed, the ongoing AI race with China is brought up—by President Donald Trump, Treasury Secretary Scott Bessent, and even AI leaders like Sam Altman and Jensen Huang
- There’s a reason for her questioning: The bills before the council would bar anyone from selling or deploying an AI system in the city unless an outside validator had checked it and a human
- Kokotajlo, now executive director of the AI Futures Project, said the labs’ ability to spot misaligned AI is poor and getting worse
Summary
“On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” said Coxon, who resigned from Anthropic in September after saying AI companies were “gambling” with human lives. Coxon voluntarily testified, and was joined by former OpenAI researcher Daniel Kokotajlo, who testified under subpoena that the companies may not know when their safety work has failed. “Our ability to even notice misalignment problems is already poor and is set to get much worse soon,” Kokotajlo testified.