{"version":1,"type":"story","url":"https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o","json":"https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o.json","markdown":"https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o.md","slug":"researchers-propose-new-framework-for-training-llm-agents-with-mixed-o","headline":"Researchers propose new framework for training LLM agents with mixed On-Policy and Off-Policy data","summary":"A new paper on arXiv argues that training interactive LLM agents requires reconsidering how on-policy and off-policy distillation are applied. The authors introduce a hierarchical distillation framework that separates the levels at which the student model follows the teacher’s policy—one for generating interaction histories and another for producing action prefixes. Their study across three models and four environments shows that pairing student-induced histories with teacher-generated prefixes yields the strongest standalone results, while fully on-policy training performs worst alone. However, mixing configurations—especially front-loading teacher-generated contexts—can improve performance, suggesting nuanced control over distillation levels matters more than raw data proportions.\n\nThe work focuses on self-distillation, where a frozen teacher model provides guidance unavailable to the student. Findings indicate that performance varies significantly based on how the two levels of on-policiness are paired and scheduled, offering practical guidelines for optimizing agent training without relying solely on either extreme of on-policy or off-policy methods.","keyPoints":["New hierarchical distillation framework separates interaction history generation from action prefix generation in LLM agents","Study finds student-induced histories with teacher-generated prefixes perform best, while fully on-policy training lags alone","Mixtures of configurations—especially front-loading teacher contexts—outperform static on-policy/off-policy splits"],"whyItMatters":"This research could reshape how AI agents are trained, offering a more flexible approach to balancing teacher and student policies. It may reduce reliance on costly on-policy data while improving agent performance in real-world tasks.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":[],"people":[]},"firstPublishedAt":"2026-10-26T04:00:00Z","updatedAt":"2026-10-26T04:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.AI","title":"Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?","url":"https://arxiv.org/abs/2610.31918","publishedAt":"2026-10-26T04:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Researchers propose new framework for training LLM agents with mixed On-Policy and Off-Policy data\", 26 October 2026, https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o","publisher":"Digest AI","title":"Researchers propose new framework for training LLM agents with mixed On-Policy and Off-Policy data","datePublished":"2026-10-26T04:00:00Z","url":"https://digestai.news/story/researchers-propose-new-framework-for-training-llm-agents-with-mixed-o"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}