Researchers propose new framework for training LLM agents with mixed On-Policy and Off-Policy data
A new paper on arXiv argues that training interactive LLM agents requires reconsidering how on-policy and off-policy distillation are applied. The authors introduce a hierarchical distillation framework that separates the levels at which the student model follows the teacher’s policy—one for generating interaction histories and another for producing action prefixes. Their study across three…
Key points
- New hierarchical distillation framework separates interaction history generation from action prefix generation in LLM agents
- Study finds student-induced histories with teacher-generated prefixes perform best, while fully on-policy training lags alone
- Mixtures of configurations—especially front-loading teacher contexts—outperform static on-policy/off-policy splits
The work focuses on self-distillation, where a frozen teacher model provides guidance unavailable to the student. Findings indicate that performance varies significantly based on how the two levels of on-policiness are paired and scheduled, offering practical guidelines for optimizing agent training without relying solely on either extreme of on-policy or off-policy methods.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers release LAMB, a Latin language model trained on 74B tokens · 1 src
- Researchers explore how aligned LLMs generalize to new tasks · 1 src
- Researchers propose agentic framework for emotion reasoning with evidence checks · 1 src
- Researchers propose 24 quality dimensions for knowledge graphs used by LLMs · 1 src
- Researchers introduce Schema Coherence Inventory to test belief consistency in language models · 1 src
Comments
via GitHub Discussions