{"version":1,"type":"story","url":"https://digestai.news/story/coding-agent-harnesses-triple-token-costs-without-boosting-benchmark-s","json":"https://digestai.news/story/coding-agent-harnesses-triple-token-costs-without-boosting-benchmark-s.json","markdown":"https://digestai.news/story/coding-agent-harnesses-triple-token-costs-without-boosting-benchmark-s.md","slug":"coding-agent-harnesses-triple-token-costs-without-boosting-benchmark-s","headline":"Coding agent harnesses triple token costs without boosting benchmark scores, study finds","summary":"Complex harnesses for coding agents inflate token bills by up to three times while barely improving benchmark performance over minimal setups, according to a study by researchers Liu and Han at Shandong University. The study evaluated five models—including Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview—across three harnesses: Claude Code, OpenCode, and mini-SWE-agent on SWE-bench Verified tasks.\n\nTesting on a pool of 447 tasks revealed that Claude Code and mini-SWE-agent performed within two points of each other, falling inside the margin of statistical noise. Swapping harnesses altered median task outcomes by 13 percent, exactly matching the variance observed from simply rerunning the same setup twice. The only major performance outlier was OpenCode, which trailed by up to nine points due to strict output cutoffs and tighter execution time limits.\n\nCost variations stemmed primarily from static system prompts and tool descriptions resent on every step. Claude Code consumed roughly 16,600 prompt tokens per step, compared to around 7,000 for OpenCode and under 1,000 for mini-SWE-agent. The researchers noted that while tool additions did not lift scores, thinking modes did prevent double-digit score drops on select models.","keyPoints":["Switching harnesses changed task outcomes by 13 percent, matching the 13 percent variance seen across identical reruns.","Harness choices tripled single-task costs, driven by system prompts spanning 16,600 tokens in Claude Code versus under 1,000 in mini-SWE-agent.","Claude Code and mini-SWE-agent finished within two points of each other on 447 SWE-bench Verified tasks."],"whyItMatters":"The findings challenge claims that sophisticated agent frameworks drive coding performance, indicating engineering efforts may inflate inference costs without delivering genuine benchmark gains.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":["Shandong University","Zhipu AI","Nvidia"],"models":["Claude Code","OpenCode","mini-SWE-agent","Claude Opus 5","Deepseek-V4-Flash","Qwen3.6"],"people":["Liu","Han"]},"firstPublishedAt":"2026-10-24T09:21:39Z","updatedAt":"2026-10-24T09:21:39Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"The Decoder","title":"Fancy harnesses make coding agents more expensive but not better, study finds","url":"https://the-decoder.com/fancy-harnesses-make-coding-agents-more-expensive-but-not-better-study-finds","publishedAt":"2026-10-24T09:21:39Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":{"title":"Google Unveils Harness Framework for Self Improving Agents","url":"https://digestai.news/thread/google-research-open-sources-rrsi-for-self-improving-ai-agents-without","storyCount":5},"cite":{"text":"Digest AI, \"Coding agent harnesses triple token costs without boosting benchmark scores, study finds\", 24 October 2026, https://digestai.news/story/coding-agent-harnesses-triple-token-costs-without-boosting-benchmark-s","publisher":"Digest AI","title":"Coding agent harnesses triple token costs without boosting benchmark scores, study finds","datePublished":"2026-10-24T09:21:39Z","url":"https://digestai.news/story/coding-agent-harnesses-triple-token-costs-without-boosting-benchmark-s"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}