Google researchers find way to prevent self-improving AI agents from memorizing tests
Self‑improving AI agents risk memorizing the tests they run, which can hurt performance on new tasks. A new paper from Google Cloud AI Research and several universities proposes Regularized Recursive Self‑Improvement of Agent Harnesses (RRSI) to curb this effect while also cutting compute costs.
Key points
- RRSI boosts unseen benchmark scores by up to 4.7 points while cutting token use by 30%
- RRSI harness uses Claude Opus 4.8 frozen model and outperforms baseline on 8 benchmarks
- Harness optimization with Gemini 3.5 Flash raised Gemini 3.1 Flash Lite accuracy from 11.2 to 14.6
RRSI adds a budget that limits how many edits a candidate can bundle, shrinks that budget over time, and uses a critic to reject benchmark‑specific changes. In experiments on eight benchmarks, the method raised scores on training tasks by up to 14.1 points and on unseen tasks by up to 4.7 points, while using about 30 % fewer tokens at runtime.
The study also shows that harnesses optimized on one frozen model can help weaker models; a Gemini 3.5 Flash harness improved Gemini 3.1 Flash Lite accuracy from 11.2 to 14.6. Nvidia’s SoL‑Pi and Google’s prior “dream” work illustrate similar gains, suggesting harness design is a key lever for safe, efficient agents.
The story so far
6 episodes →- Google researchers find way to prevent self-improving AI agents from memorizing teststhis story
Google researchers find a way to keep self-improving AI agents from memorizing their tests
The Decoder · 4 October 2026
Loading the full article…
This text was published by The Decoder and written by Jonathan Kemper. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Gemini may delete existing text when asked to append, author reports · 1 src
- Google's Gemini outperforms ChatGPT and Claude as Android assistant · 1 src
- Meta launches Muse AI agent in Mexico, first in Latin America · 11 src
- Meta rolls out real-time voice and expressive avatars for Muse AI in the US · 7 src
- Meta improves Muse and Meta AI with Active Memory and Active Focus · 2 src
Comments
via GitHub Discussions