{"version":1,"type":"story","url":"https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks","json":"https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks.json","markdown":"https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks.md","slug":"harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks","headline":"HarnessDev Finds LLM‑Built Agent Harnesses Strong in Writing, Weak in Code Tasks","summary":"HarnessDev, a new evaluation framework from researchers at ByteDance Seed, Singapore University of Technology and Design, Georgia Tech, M‑A‑P and TokenWave.AI, flips the usual benchmark focus: it scores the agent harness – the execution loop, tools and verification code – that a large language model generates, rather than the model’s final answer. Six creator LLMs (Opus 4.8, GPT‑5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max and Seed 2.0 Pro) built harnesses across four domains and five benchmarks, totaling 2,207 test instances.\n\nIn self‑evaluation, Opus 4.8 achieved the highest average score (67.8) and excelled on writing (EQ‑Bench3) and ML experimentation (MLE‑bench), but all models lagged on code‑heavy tasks such as SWE‑Pro and search‑oriented BrowseComp. Switching the executor to Gemini reshuffled rankings, causing Opus 4.8’s SWE‑Pro score to drop dramatically. Evolutionary updates yielded only modest improvements: 34 of 64 changes aligned with held‑out task gains, and many generated state‑or memory components never executed. The study highlights that harness quality is tightly coupled to the execution environment and that current LLM‑generated agents still produce substantial dead code.","keyPoints":["HarnessDev scores the runnable harness code an LLM creates, not the model's final answer","Opus 4.8 leads on writing and ML benchmarks but its SWE‑Pro score falls from 69.3 to 33.0 with a Gemini executor","Only 34 of 64 evolution changes improve held‑out tasks, indicating modest, noisy gains"],"whyItMatters":"Understanding how LLMs construct their own execution harnesses reveals bottlenecks in autonomous AI agents, guiding future research toward more reliable, executor‑agnostic tool use and reducing wasted compute on dead code.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":["ByteDance Seed","Singapore University of Technology and Design","Georgia Institute of Technology","M‑A‑P","TokenWave.AI"],"models":["Opus 4.8","GPT-5.5","Gemini 3.1 Pro","DeepSeek V4 Pro","Qwen 3.7 Max","Seed 2.0 Pro"],"people":[]},"firstPublishedAt":"2026-09-11T22:01:08Z","updatedAt":"2026-09-14T04:00:00Z","sourceCount":2,"hasPrimarySource":true,"sources":[{"outlet":"MarkTechPost","title":"Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize","url":"https://marktechpost.com/2026/09/11/can-llms-engineer-their-own-agent-harness-bytedance-seeds-harnessdev-says-only-34-of-64-changes-generalize","publishedAt":"2026-09-11T22:01:08Z","type":"press","primary":false,"lead":true},{"outlet":"arXiv cs.AI","title":"Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite","url":"https://arxiv.org/abs/2609.11987","publishedAt":"2026-09-14T04:00:00Z","type":"primary","primary":true,"lead":false}],"sourceNotes":null,"discussions":[],"thread":{"title":"AI Arms Race Meets Safety Hurdles","url":"https://digestai.news/thread/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks","storyCount":6},"cite":{"text":"Digest AI, \"HarnessDev Finds LLM‑Built Agent Harnesses Strong in Writing, Weak in Code Tasks\", 11 September 2026, https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks","publisher":"Digest AI","title":"HarnessDev Finds LLM‑Built Agent Harnesses Strong in Writing, Weak in Code Tasks","datePublished":"2026-09-11T22:01:08Z","url":"https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}