GPT · Hugging Face
On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
Most benchmarks report the first number.
Key facts
- A ReAct agent backed by GPT-4.1 posts a Mean@5 of 77.4%, genuinely strong
- Aggregate Pass^5 rises 53.0% → 69.0% while Mean@5 rises 77.4% → 81.0%, narrowing the gap between "looks capable" and "can be counted on" from 24.4pp to 12.0pp
- Medium +22.9pp (+44% relative), Hard +14.3pp (+45% relative), effectively tied in relative terms, with Medium ahead absolutely
- The team evaluated on AppWorld test_normal (168 tasks) with a ReAct agent on GPT-4.1, generating consistency guidelines from a single baseline trajectory per task and testing them on 5 fresh runs
Summary
That is embarrassing onstage. Most benchmarks hide this variability behind an average. In an earlier post, they introduced ALTK-Evolve, a system that turns an agent's own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. Accuracy hides an unreliability problem. A ReAct agent (GPT-4.1 on AppWorld test_normal ) that succeeds 77.4% of the time on average succeeds on all 5 repeated runs for only 53.0% of tasks, a 24.4-point consistency gap. The team built a diagnostic for exactly this. The Consistency Analyzer resamples an agent's own recorded trajectory to find flip-prone decision points, steps where the model was one token-sample away from doing something different.