Mixture of experts reshapes inference compute and data movement pipelines
Mixture‑of‑Experts (MoE) models have altered the architecture of inference services beyond simply adding parameters. Inference now runs in a cluster managed by orchestration layers such as NVIDIA Dynamo or Mooncake, which feed requests to servers like vLLM or SQlang. The workflow is broken into four regimes – prefill, midfill, decode attention, and decode experts – each with its own…
Key points
- MoE inference splits into prefill, midfill, decode attention, and decode experts, each with distinct compute‑memory intensity.
- KV cache is stored as immutable blobs that move across HBM, CPU DRAM, and SSD tiers to minimize data movement.
- Orchestration platforms (e.g., NVIDIA Dynamo, Mooncake) and careful batching are essential for scaling multi‑trillion‑parameter MoE models.
The KV cache is treated as immutable blobs that move across storage tiers: active prefixes reside briefly in accelerator HBM, then shift to CPU DRAM, SSD, or shared network‑attached memory. Efficient batching and routing of tokens to shared experts reduce weight loading, but require tight networking, especially when thousands of experts are spread across many GPUs. The essay highlights trade‑offs between aggregating stages on a single node versus disaggregating them across racks, and notes that future memory technologies such as 3‑D RAM will be decisive for handling the multi‑hundred‑kilobyte contexts now common in frontier MoE models.
The story so far
2 episodes →- Mixture of experts reshapes inference compute and data movement pipelinesthis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Hardware & Compute
All →- Jeff Dean says automation could cut chip design teams to ten and cycles to three months · 1 src
- Nvidia launches DSX Ready program to qualify power and cooling products · 1 src
- Nvidia introduces Halos safety system for autonomous vehicles and robots · 1 src
- Apple and Tesla were better insulated from Hugging Face breach, analysts say · 1 src
- Google launches Googlebook laptops with Android‑ChromeOS hybrid · 3 src
Comments
via GitHub Discussions