GitHub · Agentic AI · GitHub Blog
ReviewBench: An open benchmark for AI code review
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
★ Tier-1 Source
Agentic code review is becoming an essential piece of how development happens.
Key facts
- ReviewBench contains 219 pull requests from 187 public open source licensed repositories spanning 19 languages, with its language and repository-size distributions closely matching GitHub overall
- The online A/B test moved in the same direction as ReviewBench predicted: addressed rate (precision) rose 8.0%, recall rose 13.6%, and comment volume rose 61%, while cost per review fell 8.0%, all
- 219 public pull requests across 19 languages, aligned to GitHub-wide distributions while preserving substantive review cases
- ReviewBench’s severity-level evaluation captured this too: it predicted a 227% increase in critical comments, compared with 262% online, along with the same broader shift toward more moderate
Summary
But the quality of existing AI reviewers can be hard to measure, and you need to know the strengths of a reviewer before you know if it will help you. That makes it important to understand how reviewers compare: what different systems catch, what they miss, and the tradeoffs they make. The team built ReviewBench, a new code review offline benchmark, to address that gap, and it is available for you to use today. In this post, they'll walk through how ReviewBench is constructed, how it establishes reliable ground truth and scoring, and how to onboard your own code review system and submit results.