New paper outlines emerging trends in AI agent evaluation

The paper highlights rising costs, noting that evaluating several agents on the ExploitBench benchmark required tens of thousands of dollars in credits, not counting the engineering effort to maintain infrastructure. It also points to the METR randomized controlled trial, which found that using agent tools for software development increased overall task completion time. The authors argue that evaluation must move beyond final outcomes to examine entire trajectories, shortcuts, and tool access, and that independent evaluators need funding and model access to remain effective.

Recommendations include expanding interdisciplinary participation, improving benchmark rigor, and fostering transparent, independent evaluation practices. The authors see a window to build a higher‑standard field for AI agent evaluation that can inform labs, policymakers, and industry deployments.