GPT · AI Agent · withspecific.com
Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Benchmarking frontier AI models on private, real-world, enterprise codebases.
Key facts
- 71.4 % of rollouts under 10 minutes failed, compared with 73.4 % of longer rollouts
- This leads to their prompts being slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions
- Can a coding agent do the work of a software engineer in the real world
- Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need
Summary
Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet. Work with business consequences. Getting billing right, calculating taxes, migrating customers. Company-specific complexity. Agents have to understand those conventions and make changes that work with what’s already there. Can a coding agent do the work of a software engineer in the real world?