HarnessDev Finds LLM‑Built Agent Harnesses Strong in Writing, Weak in Code Tasks

In self‑evaluation, Opus 4.8 achieved the highest average score (67.8) and excelled on writing (EQ‑Bench3) and ML experimentation (MLE‑bench), but all models lagged on code‑heavy tasks such as SWE‑Pro and search‑oriented BrowseComp. Switching the executor to Gemini reshuffled rankings, causing Opus 4.8’s SWE‑Pro score to drop dramatically. Evolutionary updates yielded only modest improvements: 34 of 64 changes aligned with held‑out task gains, and many generated state‑or memory components never executed. The study highlights that harness quality is tightly coupled to the execution environment and that current LLM‑generated agents still produce substantial dead code.