# Researchers propose new framework for training LLM agents with mixed On-Policy and Off-Policy data

Digest AI · Research · published 2026-10-26T04:00:00Z

Canonical: https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o

## Summary

A new paper on arXiv argues that training interactive LLM agents requires reconsidering how on-policy and off-policy distillation are applied. The authors introduce a hierarchical distillation framework that separates the levels at which the student model follows the teacher’s policy—one for generating interaction histories and another for producing action prefixes. Their study across three models and four environments shows that pairing student-induced histories with teacher-generated prefixes yields the strongest standalone results, while fully on-policy training performs worst alone. However, mixing configurations—especially front-loading teacher-generated contexts—can improve performance, suggesting nuanced control over distillation levels matters more than raw data proportions.

The work focuses on self-distillation, where a frozen teacher model provides guidance unavailable to the student. Findings indicate that performance varies significantly based on how the two levels of on-policiness are paired and scheduled, offering practical guidelines for optimizing agent training without relying solely on either extreme of on-policy or off-policy methods.

## Key points

- New hierarchical distillation framework separates interaction history generation from action prefix generation in LLM agents
- Study finds student-induced histories with teacher-generated prefixes perform best, while fully on-policy training lags alone
- Mixtures of configurations—especially front-loading teacher contexts—outperform static on-policy/off-policy splits

## Why it matters

This research could reshape how AI agents are trained, offering a more flexible approach to balancing teacher and student policies. It may reduce reliance on costly on-policy data while improving agent performance in real-world tasks.

## Sources

1. [Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?](https://arxiv.org/abs/2610.31918) (arXiv cs.AI, 2026-10-26, primary source)

## Cite

Digest AI, "Researchers propose new framework for training LLM agents with mixed On-Policy and Off-Policy data", 26 October 2026, https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o.json