StrongREJECT · Hugging Face
BenchMIRT found that WMDP scores were more strongly associated with general reasoning than with safety
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
HarmBench, which tests whether models comply with harmful requests, shows how a single benchmark can mix together different kinds of signal.
Key facts
- The team trained BenchMIRT on benchmarking results from 100 LLMs across 16 benchmarks and more than 34K questions
- The other 10 come from their Olmo 3 safety suite, including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest
- Bar size and direction show the Pearson correlation, across 100 open-weight LLMs, between BenchMIRT ability scores and benchmark scores on a −1 to 1 scale—pink for general reasoning and teal
- In their experiments, it correctly predicted whether a model would answer a held-out question correctly 79% of the time
Summary
Today they're introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts—the questions and tasks a model is scored on. A benchmark is usually designed to measure a particular ability, such as safety, general reasoning, or instruction following. Take BBQ, a benchmark designed to test whether models rely on social stereotypes. And even within a single benchmark, different groups of questions and tasks can measure different things. BenchMIRT helps researchers separate those signals and see what’s driving a benchmark’s score. BenchMIRT takes cues from Item Response Theory (IRT), a technique originating in psychometrics—the field concerned with measuring abilities and traits from patterns of test responses.