Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Olmo-core 3 addresses this by redesigning the training stack to minimize communication costs. In benchmarks, it increased the expert pool from 8 to 128 while keeping active parameters per token near 3.2B, growing total capacity from 4.6B to 47B parameters. Training throughput dropped by less than 5% compared to earlier versions. The framework also supports one trillion total parameters in tests, with optimizations like expert parallelism, pipeline parallelism, and distributed optimizers to distribute workloads across GPUs. A preliminary test on eight NVIDIA B300 GPUs showed a 47-billion-parameter MoE processed 52,000 tokens per second per GPU, a 2.7× improvement over the previous implementation.

The release includes techniques like rowwise expert parallelism, GPU-resident routing, and grouped GEMM to reduce data movement and computation overhead. It also supports MXFP8, a lower-precision format, which increased training throughput by 21% in controlled tests while reducing peak memory usage from 103 GiB to 95 GiB. The framework is fully open-source, allowing researchers to adapt it for different hardware and experiments.