Continual Search framework boosts AI agent failure attribution accuracy on large logs
The paper tackles the challenge of diagnosing failures in long‑horizon AI agents, where execution logs can span thousands of steps and human review is infeasible. Existing root‑cause attribution (RCA) methods that rely on a single LLM judgment often stop early, missing sparse evidence that is distributed across the trace and resulting in low diagnostic accuracy.
Key points
- Existing LLM‑based RCA methods miss evidence in long execution traces, leading to low diagnostic accuracy.
- Continual Search iteratively nudges the model to keep searching, improving attribution across four benchmarks and the new MegaRCA‑Mix.
- On MegaRCA‑Mix, GPT‑5.5’s F1 rises from 0.349 to 0.498, a >40% gain, letting smaller models beat larger ones.
To address this, the authors introduce Continual Search, an iterative framework that repeatedly prompts the LLM judge to keep looking for unresolved evidence. They evaluate the approach on four established RCA benchmarks and on a newly created MegaRCA‑Mix suite containing 50 human‑annotated, execution‑heavy failure cases. The method consistently improves performance; for example, GPT‑5.5’s F1 score on MegaRCA‑Mix rises from 0.349 to 0.498, a gain of over 40 %. The results also show that effective search can let smaller models outperform larger ones.
The study highlights that systematic search strategies can be more critical than raw model scale for reliable agent diagnostics, pointing to a promising direction for building trustworthy autonomous systems.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Research
All →- AI Researchers Fear Machines Could Kill Us All · 9 src
- Token merging boosts Whisper efficiency across 16 languages with minimal accuracy loss · 1 src
- LabAgent: Automates Reproducing Scientific Methods · 1 src
- Vibe Patenting: LLM Judges Improve AI Patent Drafting Quality · 1 src
- Generalized Agent Iteration Framework Unifies Policy Improvement and Recursive Self-Improvement · 1 src
Comments
via GitHub Discussions