Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
Researchers introduced Agent4Sci, a long‑horizon testbed that evaluates how current AI coding agents can enhance published best models on scientific machine‑learning benchmarks. The suite covers 20 benchmarks across 11 domains and measures progress under in‑the‑wild conditions rather than a leaderboard ranking. In the baseline runs, frontier agents advanced the state‑of‑the‑art on every…
Key points
- Frontier coding agents improve published best models on all 20 benchmarks; Claude Opus achieves 10.4% mean relative improvement.
- Adding lightweight plug‑in skills raises Claude Opus to 20.1% and Codex to 8.1% mean improvement.
- Gains come mainly from ML‑engineering tricks; agents lack research‑level insight and face novelty miscalibration, resource under‑utilization, and local sweep collapse.
When the authors equipped the agents with lightweight plug‑in skills for research ideation, experiment management, and scheduled checklists, performance rose sharply: Claude Opus reached a 20.1% mean improvement and Codex climbed to 8.1%. Analysis of the trajectories revealed three recurring shortcomings – novelty miscalibration, resource under‑utilization, and rapid collapse into local engineering sweeps – indicating that current gains stem largely from generic ML‑engineering tricks rather than genuine scientific insight. The authors release the code and data on GitHub and Hugging Face to let the community track and push agents toward deeper domain understanding.
Agent4Sci thus provides a concrete platform for measuring and advancing the capabilities of AI agents on real‑world ML‑for‑science problems.
The story so far
2 episodes →- Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?this story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- PHP patches critical RCE flaws discovered with GPT-5.6 Sol, affecting up to 70% of websites · 9 src
- Researchers release madartb, a parallel Multi-Dialect Arabic dependency treebank · 1 src
- Researchers decode 26 legal factors in three open LLMs · 1 src
- Researchers test how reinforcement learning teaches AI agents to enforce market collusion · 1 src
- LLMs can assess psychotherapy fidelity, study shows · 1 src
Comments
via GitHub Discussions