{"version":1,"type":"story","url":"https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm","json":"https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm.json","markdown":"https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm.md","slug":"researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm","headline":"Researchers propose Li²SAO to improve asynchronous RL training for LLMs","summary":"A new paper on arXiv introduces **Li²SAO**, a method to enhance asynchronous reinforcement learning (RL) for training large language models (LLMs). The approach builds on **LiSAO**, which previously struggled with stability due to reliance on a closed-source value model. Li²SAO uses **group-relative actor-critic bootstrapping** to train the actor and critic more effectively, especially in early training stages, and later switches to single-rollout GAE for improved performance.\n\nThe authors compare **Li²SAO** with **GRPO** (Group-Relative Policy Optimization) and **LiSAO** across single-turn reasoning and multi-turn agentic tasks. GRPO performs better on reasoning tasks, while Li²SAO excels on agentic tasks. The paper notes that LiSAO’s early training lags due to an untrained value model, but Li²SAO addresses this by dynamically adjusting training phases. The code is publicly available on GitHub.","keyPoints":["Li²SAO improves asynchronous RL training for LLMs by using group-relative actor-critic bootstrapping","GRPO outperforms Li²SAO on reasoning tasks, but Li²SAO surpasses it on agentic tasks","Early training instability in LiSAO is mitigated by Li²SAO’s phased training approach"],"whyItMatters":"This method could accelerate RL-based LLM training by reducing reliance on pretrained value models, making asynchronous optimization more practical for complex tasks.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":["GRPO","LiSAO","Li²SAO"],"people":[]},"firstPublishedAt":"2026-11-02T05:00:00Z","updatedAt":"2026-11-02T05:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.CL","title":"Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping","url":"https://arxiv.org/abs/2610.40726","publishedAt":"2026-11-02T05:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Researchers propose Li²SAO to improve asynchronous RL training for LLMs\", 2 November 2026, https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm","publisher":"Digest AI","title":"Researchers propose Li²SAO to improve asynchronous RL training for LLMs","datePublished":"2026-11-02T05:00:00Z","url":"https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}