Researchers introduce RegretBench and BackUpper for coding agents
Researchers from arXiv have introduced RegretBench, a benchmark designed to test the "backup awareness" of autonomous coding agents. The benchmark consists of 205 executable tasks derived from real-world regret moments reported on platforms like X.com, GitHub, and Stack Overflow, where users explicitly stated they needed state they had failed to preserve. The study evaluates six frontier models,…
Key points
- RegretBench contains 205 tasks based on real user regrets about lost state.
- Frontier agents complete 86.1% of tasks but back up state in only 15.2% of cases.
- BackUpper restores state with 92.2% similarity at a cost of $0.014 per task.
To address this limitation, the authors propose BackUpper, a harness plug-in that acts as a backup layer. It intercepts every tool call to preserve affected state before modification, using rule-based strategies for common operations and a lightweight subagent for complex scenarios. On the RegretBench, BackUpper covers every regret moment action and restores states with 92.2% average similarity to the original. The system achieves this at a cost of $0.014 per task, using 7x fewer tokens than a main-agent self-backup baseline. Additionally, when user preferences are provided, BackUpper matches human annotations on preservation worthiness with an 84.5% F1 score, suggesting agents can maintain reversibility at low cost.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- PHP patches critical RCE flaws discovered with GPT-5.6 Sol, affecting up to 70% of websites · 9 src
- OpenAI and Google AI models discovered critical PHP flaw affecting millions of websites · 8 src
- Researchers release madartb, a parallel Multi-Dialect Arabic dependency treebank · 1 src
- Researchers decode 26 legal factors in three open LLMs · 1 src
- Researchers test how reinforcement learning teaches AI agents to enforce market collusion · 1 src
Comments
via GitHub Discussions