Researchers propose Li²SAO to improve asynchronous RL training for LLMs
A new paper on arXiv introduces Li²SAO, a method to enhance asynchronous reinforcement learning (RL) for training large language models (LLMs). The approach builds on LiSAO, which previously struggled with stability due to reliance on a closed-source value model. Li²SAO uses group-relative actor-critic bootstrapping to train the actor and critic more effectively, especially in early training…
Key points
- Li²SAO improves asynchronous RL training for LLMs by using group-relative actor-critic bootstrapping
- GRPO outperforms Li²SAO on reasoning tasks, but Li²SAO surpasses it on agentic tasks
- Early training instability in LiSAO is mitigated by Li²SAO’s phased training approach
The authors compare Li²SAO with GRPO (Group-Relative Policy Optimization) and LiSAO across single-turn reasoning and multi-turn agentic tasks. GRPO performs better on reasoning tasks, while Li²SAO excels on agentic tasks. The paper notes that LiSAO’s early training lags due to an untrained value model, but Li²SAO addresses this by dynamically adjusting training phases. The code is publicly available on GitHub.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- SemiAnalysis releases GDPx benchmark with Claude Opus 5.5 leading at 12% · 3 src
- Gemma-3-12b fails to detect misinformation when poisoned claims dominate retrieved evidence · 1 src
- Researchers release BILAL-BENCH to test Bengali AI models’ Islamic text fabrication · 1 src
- Researchers prune 729 attention heads to cut GPT-2-XL’s memorisation by 91.5% · 1 src
- arXiv paper finds shortcuts distort RAG benchmarking accuracy · 1 src
Comments
via GitHub Discussions