Recursive language models generalize out of domain, study shows
The study examines how limiting a language model's view can affect learning. It compares chain‑of‑thought (CoT) with recursive language models that solve each subtask in isolation.
Key points
- Recursive language models isolate subtasks, preventing context leakage that can cause shortcut learning.
- Chain‑of‑thought can overfit to out‑of‑domain tokens, leading to failures when those tokens change.
- In‑distribution performance is similar, but out‑of‑domain generalization favors recursive models.
In‑distribution, both approaches perform similarly; CoT can simulate the recursive rule, so generalization differs only by a constant factor. However, out‑of‑domain, CoT tends to fit training data by relying on context outside the current subtask, creating a shortcut that fails when those tokens change. Recursive isolation prevents this failure mode.
The authors argue that covering the right rule is insufficient for true reasoning; simplicity bias leads the model to choose shortcuts over correct reasoning. This contrasts with classical learning theory and highlights the need for context isolation to improve out‑of‑domain robustness.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Research
All →- Study finds token-level entropy blind in small language models, semantic entropy helps · 1 src
- Researchers propose AI-GRACE framework for operationalizing agentic AI use cases · 1 src
- Embed‑TTT improves rule induction in ARC‑like tasks, authors say · 1 src
- Researchers introduce SpecOpt for agentic molecule specificity optimization · 1 src
- researchers report agent-based hls with rtl refinement speeds chip design 2.6× · 1 src
Comments
via GitHub Discussions