California · Anthropic · OpenAI · Dario Amodei · Sam Altman · Donald Trump · CNBC Technology
METR, a nonprofit, announced in August that it had raised commitments of around $71 million over the last six months
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
By late that month, METR's profile had risen further.
Key facts
- Rayan Krishnan, CEO of independent evaluator Vals AI, said his for-profit startup, which builds benchmarks to measure how AI models perform on industry-specific tasks, has grown from eight employees
- METR, a nonprofit, announced in August that it had raised commitments of around $71 million over the last six months
- The AI Evaluator Forum, which includes METR, the AI Verification and Evaluation Research Institute (AVERI), and other groups, published a public letter last month titled, "Minimum Conditions
- Anthropic CEO Dario Amodei pledged to embed independent evaluators in his company last month, a move that OpenAI CEO Sam Altman quickly endorsed
Summary
Two months ago, independent evaluators occupied a relatively sleepy corner of the multitrillion-dollar artificial intelligence industry. While Anthropic and OpenAI are at the heart of a fierce debate over whether they can safeguard their advanced models and grow their businesses simultaneously, they're seeking support from a handful of small third-party groups like Model Evaluation and Threat Research (METR), Apollo Research, and Transluce. The evaluators, which mostly operate as nonprofits, are still finding their footing in an industry where capital is flowing at historic levels and new models are rolling out faster than ever. Without a federal push for regulation, evaluators have taken on outsized importance. "To a degree, the problem, as always, is money," Suresh Venkatasubramanian, a computer science professor at Brown University, told CNBC in an interview.