Researchers release Dory1.0-4B, a 4B-parameter looped model with 256K-token context
Researchers introduced Dory1.0-4B, a 4-billion-parameter language model with a looped architecture that adjusts compute budgets dynamically. The model uses a normalized Transformer (nGPT) backbone with sliding-window and full-attention layers, split into input, recurrent, and output blocks. During inference, the recurrent block can loop multiple times, improving accuracy without increasing…
Key points
- Dory1.0-4B is a 4B-parameter looped model with 256K-token context, using a normalized Transformer architecture
- Pre-trained on 6T tokens (4.5T in NVFP4), then fine-tuned with SFT and RLVR for looped inference
- Outperforms larger models like Qwen3.5-9B and Gemma 4 on coding/agentic tasks, with 1.3–2.5x token efficiency gains
Dory1.0-4B was pre-trained on 6 trillion tokens, with the first 4.5 trillion trained in NVFP4 precision. Post-training included supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), using a difficulty-aware loop curriculum. The model outperforms larger models like Qwen3.5-9B and Gemma 4 12B/26B-A4B on coding, agentic, and long-context benchmarks. Looping also improves token efficiency, reducing output by 1.3–2.5x on agentic tasks. The model’s checkpoints are open-sourced in BF16, MXFP8, NVFP4, and GGUF formats.
Model page: Gemma 4 26B A4B →
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Tencent releases WeVisOne-VL, a 2B/4B open vision model for detection, OCR and matting · 1 src
- Anthropic releases FlashProve open-source system for provable AI inference · 6 src
- Lama-dt outperforms baselines on meta-world and mujoco · 1 src
- SLATE benchmark tests long-horizon API selection in e-commerce automation · 1 src
- Researchers compare two fine-tuning methods for small medical AI models · 1 src
Comments
via GitHub Discussions