NCP-ArchPreview: Latent Space Language Model Advances
Researchers have unveiled NCP-ArchPreview, a latent-space language model that extends beyond standard next-token prediction (NTP) by incorporating Next Concept Prediction (NCP). This model learns to predict discrete concepts spanning multiple tokens, significantly challenging autoregressive pretraining. By training on the Dolma-3 dataset with 5.73T tokens and scaling up to 8.9B parameters,…
1 source primary source
Key points
- NCP-ArchPreview uses Next Concept Prediction (NCP) to learn discrete concepts spanning multiple tokens
- Trains on the Dolma-3 dataset with 5.73T tokens and reaches up to 8.9B parameters
- Improves downstream tasks by 2.45 points compared to OLMo-3-7B
Read the original at arXiv cs.CL · by NCP Team, Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge, Ruijun Ge, Jiayi Huang, Yang Jiao, Dahua Lin, Zhouhan Lin, Yifan Liu, Yuliang Liu, Biqing Qi, Mowen Ruan, Junzhe Shen, Yunchong Song, Hao Sun, Zhongbo Tian, Yixuan Wang, Rubin Wei, Jiaxin Xiong, Kangyu Yang, Qi primary sourceOpen source ↗
Topics · follow one to build your own front page
NCP-ArchPreview
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Generative AI & Models
All →- Introducing Mistral Large 4 · 7 src
- OpenAI boosts GPT-6 speed by 50% in 28 Days update · 1 src
- Reflection AI debuts Beam, a 501B open-weight model rivaling GLM 5.2 with 3-4x less compute · 10 src
- Author tests four AI assistants on flawed forecasting scenarios · 1 src
- Anthropic prompts users to share voice data to enhance AI models · 8 src
Comments
via GitHub Discussions