Prime Intellect releases Prime Mega MoE kernel for faster training
Prime Intellect has released Prime Mega MoE, a GPU kernel extension that adds a fused, overlapping backward pass to DeepSeek’s existing Mega MoE kernel. While DeepSeek’s original kernel only supported the forward pass for inference, Prime Intellect’s addition makes the fused kernel usable for full model training. The tool is designed to reduce idle GPU time caused by communication bottlenecks in…
Key points
- Prime Intellect added a fused backward pass to DeepSeek’s Mega MoE kernel for training.
- Training steps run 32% to 34% faster than the prime-rl baseline on Qwen3 models.
- Forward pass speed reaches 3.11x faster, while backward pass gains are up to 1.24x.
According to self-reported figures from Prime Intellect, the new kernel improves training efficiency significantly. On Qwen3 MoE models, full training steps run 32% to 34% faster than the prime-rl baseline. Compared to TorchTitan, the speedup reaches 48% to 55%. The most notable gain is in the forward pass, which is up to 3.11x faster on a Qwen3-30B-A3B layer, though backward pass improvements are smaller, reaching only 1.24x on larger layers.
The kernel supports BF16, MXFP8, and blockwise FP8 precision on NVIDIA Hopper and Blackwell GPUs. It is available as a dispatch backend in the prime-rl trainer, requiring a specific prebuilt DeepGEMM wheel. These performance gains translate directly into reduced GPU hours for teams training large MoE models.
Prime Intellect Releases Prime Mega MoE: Trainable Fused MoE Kernel, 3.11x Faster Forward Pass
MarkTechPost · 29 October 2026
Loading the full article…
This text was published by MarkTechPost and written by Asif Razzaq. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI expands misalignment monitors to catch agent escapes · 2 src
- Dulo raises $130M seed to build AI models for hardware design · 1 src
- Study finds voice tutors cost more but don’t prove better learning in MBA course · 1 src
- Hebrew University researchers release BBERT to classify bacterial DNA in soil samples · 1 src
- OpenAI and Anthropic disprove 27-year-old computer science hypothesis · 1 src
Comments
via GitHub Discussions