AI Agent · GitHub · Hugging Face
ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
Recent advances in coding agents have sparked excitement around AI-assisted modernization.
Key facts
- Claude Code reported successful builds for 29 out of 30 whole applications
- To address this gap, they introduce ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark for evaluating AI agents on cross-framework migration tasks in Enterprise Java
- Starting from a JSR-based enterprise Java taxonomy, expert migrations create verified implementations across Spring, Jakarta EE, and Quarkus
- Recent advances in coding agents have sparked excitement around AI-assisted modernization
Summary
Existing software engineering benchmarks have demonstrated impressive progress in bug fixing and code generation, but framework migration presents a fundamentally different challenge. To address this gap, they introduce ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark for evaluating AI agents on cross-framework migration tasks in Enterprise Java. Unlike traditional benchmarks that compare generated code against reference implementations, ScarfBench evaluates whether migrated applications build, deploy, and preserve behavior. A simple repository migration can require changes across dependency injection, persistence configuration, queries, and framework descriptors. Framework migration requires translating framework semantics, not source code.