# HarnessDev Finds LLM‑Built Agent Harnesses Strong in Writing, Weak in Code Tasks

Digest AI · Agents & Tools · published 2026-09-11T22:01:08Z · updated 2026-09-14T04:00:00Z

Canonical: https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks

## Summary

HarnessDev, a new evaluation framework from researchers at ByteDance Seed, Singapore University of Technology and Design, Georgia Tech, M‑A‑P and TokenWave.AI, flips the usual benchmark focus: it scores the agent harness – the execution loop, tools and verification code – that a large language model generates, rather than the model’s final answer. Six creator LLMs (Opus 4.8, GPT‑5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max and Seed 2.0 Pro) built harnesses across four domains and five benchmarks, totaling 2,207 test instances.

In self‑evaluation, Opus 4.8 achieved the highest average score (67.8) and excelled on writing (EQ‑Bench3) and ML experimentation (MLE‑bench), but all models lagged on code‑heavy tasks such as SWE‑Pro and search‑oriented BrowseComp. Switching the executor to Gemini reshuffled rankings, causing Opus 4.8’s SWE‑Pro score to drop dramatically. Evolutionary updates yielded only modest improvements: 34 of 64 changes aligned with held‑out task gains, and many generated state‑or memory components never executed. The study highlights that harness quality is tightly coupled to the execution environment and that current LLM‑generated agents still produce substantial dead code.

## Key points

- HarnessDev scores the runnable harness code an LLM creates, not the model's final answer
- Opus 4.8 leads on writing and ML benchmarks but its SWE‑Pro score falls from 69.3 to 33.0 with a Gemini executor
- Only 34 of 64 evolution changes improve held‑out tasks, indicating modest, noisy gains

## Why it matters

Understanding how LLMs construct their own execution harnesses reveals bottlenecks in autonomous AI agents, guiding future research toward more reliable, executor‑agnostic tool use and reducing wasted compute on dead code.

## Sources

1. [Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize](https://marktechpost.com/2026/09/11/can-llms-engineer-their-own-agent-harness-bytedance-seeds-harnessdev-says-only-34-of-64-changes-generalize) (MarkTechPost, 2026-09-11)
2. [Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite](https://arxiv.org/abs/2609.11987) (arXiv cs.AI, 2026-09-14, primary source)

Part of the developing story: [AI Arms Race Meets Safety Hurdles](https://digestai.news/thread/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks) (6 stories)

## Cite

Digest AI, "HarnessDev Finds LLM‑Built Agent Harnesses Strong in Writing, Weak in Code Tasks", 11 September 2026, https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/harnessdev-finds-llmbuilt-agent-harnesses-strong-in-writing-weak-in-code-tasks.json