Coding agent harnesses triple token costs without boosting benchmark scores, study finds

Testing on a pool of 447 tasks revealed that Claude Code and mini-SWE-agent performed within two points of each other, falling inside the margin of statistical noise. Swapping harnesses altered median task outcomes by 13 percent, exactly matching the variance observed from simply rerunning the same setup twice. The only major performance outlier was OpenCode, which trailed by up to nine points due to strict output cutoffs and tighter execution time limits.

Cost variations stemmed primarily from static system prompts and tool descriptions resent on every step. Claude Code consumed roughly 16,600 prompt tokens per step, compared to around 7,000 for OpenCode and under 1,000 for mini-SWE-agent. The researchers noted that while tool additions did not lift scores, thinking modes did prevent double-digit score drops on select models.