# Researchers propose Li²SAO to improve asynchronous RL training for LLMs

Digest AI · Research · published 2026-11-02T05:00:00Z

Canonical: https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm

## Summary

A new paper on arXiv introduces **Li²SAO**, a method to enhance asynchronous reinforcement learning (RL) for training large language models (LLMs). The approach builds on **LiSAO**, which previously struggled with stability due to reliance on a closed-source value model. Li²SAO uses **group-relative actor-critic bootstrapping** to train the actor and critic more effectively, especially in early training stages, and later switches to single-rollout GAE for improved performance.

The authors compare **Li²SAO** with **GRPO** (Group-Relative Policy Optimization) and **LiSAO** across single-turn reasoning and multi-turn agentic tasks. GRPO performs better on reasoning tasks, while Li²SAO excels on agentic tasks. The paper notes that LiSAO’s early training lags due to an untrained value model, but Li²SAO addresses this by dynamically adjusting training phases. The code is publicly available on GitHub.

## Key points

- Li²SAO improves asynchronous RL training for LLMs by using group-relative actor-critic bootstrapping
- GRPO outperforms Li²SAO on reasoning tasks, but Li²SAO surpasses it on agentic tasks
- Early training instability in LiSAO is mitigated by Li²SAO’s phased training approach

## Why it matters

This method could accelerate RL-based LLM training by reducing reliance on pretrained value models, making asynchronous optimization more practical for complex tasks.

## Sources

1. [Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping](https://arxiv.org/abs/2610.40726) (arXiv cs.CL, 2026-11-02, primary source)

## Cite

Digest AI, "Researchers propose Li²SAO to improve asynchronous RL training for LLMs", 2 November 2026, https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-propose-li2sao-to-improve-asynchronous-rl-training-for-llm.json