NCP-ArchPreview: Latent Space Language Model Advances
Researchers have unveiled NCP-ArchPreview, a latent-space language model that extends beyond standard next-token prediction (NTP) by incorporating Next Concept Prediction (NCP). This model learns to predict discrete concepts spanning multiple tokens, significantly challenging autoregressive pretraining. By training on the Dolma-3 dataset with 5.73T tokens and scaling up to 8.9B parameters,…
1 source primary source
Key points
- NCP-ArchPreview uses Next Concept Prediction (NCP) to learn discrete concepts spanning multiple tokens
- Trains on the Dolma-3 dataset with 5.73T tokens and reaches up to 8.9B parameters
- Improves downstream tasks by 2.45 points compared to OLMo-3-7B
Read the original at arXiv cs.CL · by NCP Team, Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge, Ruijun Ge, Jiayi Huang, Yang Jiao, Dahua Lin, Zhouhan Lin, Yifan Liu, Yuliang Liu, Biqing Qi, Mowen Ruan, Junzhe Shen, Yunchong Song, Hao Sun, Zhongbo Tian, Yixuan Wang, Rubin Wei, Jiaxin Xiong, Kangyu Yang, Qi primary source Open source ↗
Topics · follow one to build your own front page
NCP-ArchPreview
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Generative AI & Models
All →- OpenAI Unveils GPT‑6 Astra: Record‑Breaking 3D Rendering, Loop‑Transformer Architecture · 79 src
- Tail-Aware Scheduling Reduces Workflow Latency · 1 src
- Deterministic Math Solver Improves Clinical Calculations in LLMs · 1 src
- Multilingual Entity Linking: New Framework Improves Accuracy · 1 src
- ArrowJS: The First UI for AI Agents? · 1 src
Comments
via GitHub Discussions