GitHub releases ReviewBench to evaluate AI code reviews across 219 pull requests

The benchmark measures performance using dual metrics: grounded evaluations against a fixed reference dataset and augmented evaluations that assess newly discovered issues. Claude Sonnet 3.5 serves as the automated judgment model to match and verify feedback. An internal audit of the ground truth dataset by senior engineers matched ReviewBench judgments at 96.6%, though the materials represent benchmark specifications rather than academic peer-reviewed research.

GitHub also shared a production case study evaluating a multi-model configuration against Copilot code review's single-review setup. Offline testing predicted increases of 4.45% in precision and 12.88% in recall, alongside a 23.7% cost reduction. In production A/B testing, the addressed response rate rose by 8.00%, recall rose by 13.58%, and costs dropped by 8.00%. However, source figures conflicted regarding total comment volume, noting a 25.0% increase in a data table and 61% in body text.