Study proposes skill habits to fix AI agent inconsistency
A new arXiv paper addresses the lack of consistency in AI agents, which often produce varying results for identical tasks. The authors tested 42 tasks three times each and found that 38% to 74% of responses were inconsistent depending on the model. They also noted that 95.3% to 97.2% of generated tokens are wasted on re-deriving known plans.
Key points
- Agents showed 38% to 74% inconsistency across 42 repeated tasks in the study.
- Skill habit formation reduced token usage by 14% to 56% on text-to-SQL tasks.
- The method improved end-to-end accuracy from 0.888 to 0.952 at 43% of the cost.
The researchers propose "skill habit formation," where agents mine their own execution history to create deterministic scripts for common tasks. These scripts compete with standard reasoning, allowing the common case to run as a script while complex cases fall through to reasoning. On text-to-SQL tasks, a habit-formed variant reproduced its output on all 456 repeated dispatches, compared to 11 to 26 out of 42 for standard reasoning arms. This approach also reduced token usage by 14% to 56%.
However, the method has trade-offs. The system incorrectly admitted work it should have deferred on 2.6% of natural paraphrases and 26% of boundary inputs. Most of these failures were invisible to the system's safety checks. The authors conclude that while deterministic errors repeat exactly, separating routing from parameter extraction improved end-to-end accuracy from 0.888 to 0.952 at 43% of the cost.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Semantic Routing Calibration mitigates LLM over-refusal · 1 src
- AIBuildAI-2.5 ranks first on MLE-Bench with 73.3% medal rate · 1 src
- Researchers fine-tune 406M model for meeting summaries with retrieved text spans · 1 src
- Researchers test how language models handle numerical formats in word problems · 1 src
- ReAdapt improves warm‑introduction and reaction selection accuracy for Gemini‑3‑Flash · 1 src
Comments
via GitHub Discussions