{"version":1,"type":"story","url":"https://digestai.news/story/mixture-of-experts-reshapes-inference-compute-and-data-movement-pipeli","json":"https://digestai.news/story/mixture-of-experts-reshapes-inference-compute-and-data-movement-pipeli.json","markdown":"https://digestai.news/story/mixture-of-experts-reshapes-inference-compute-and-data-movement-pipeli.md","slug":"mixture-of-experts-reshapes-inference-compute-and-data-movement-pipeli","headline":"Mixture of experts reshapes inference compute and data movement pipelines","summary":"Mixture‑of‑Experts (MoE) models have altered the architecture of inference services beyond simply adding parameters.  Inference now runs in a cluster managed by orchestration layers such as NVIDIA Dynamo or Mooncake, which feed requests to servers like vLLM or SQlang.  The workflow is broken into four regimes – prefill, midfill, decode attention, and decode experts – each with its own compute‑to‑data‑movement profile.  Prefill and midfill enjoy higher arithmetic intensity, while decode attention and decode experts are dominated by KV‑cache reads and expert weight loads.\n\nThe KV cache is treated as immutable blobs that move across storage tiers: active prefixes reside briefly in accelerator HBM, then shift to CPU DRAM, SSD, or shared network‑attached memory.  Efficient batching and routing of tokens to shared experts reduce weight loading, but require tight networking, especially when thousands of experts are spread across many GPUs.  The essay highlights trade‑offs between aggregating stages on a single node versus disaggregating them across racks, and notes that future memory technologies such as 3‑D RAM will be decisive for handling the multi‑hundred‑kilobyte contexts now common in frontier MoE models.","keyPoints":["MoE inference splits into prefill, midfill, decode attention, and decode experts, each with distinct compute‑memory intensity.","KV cache is stored as immutable blobs that move across HBM, CPU DRAM, and SSD tiers to minimize data movement.","Orchestration platforms (e.g., NVIDIA Dynamo, Mooncake) and careful batching are essential for scaling multi‑trillion‑parameter MoE models."],"whyItMatters":"Understanding these compute and data‑movement patterns helps providers cut costs and latency when deploying large MoE models at scale.","category":{"slug":"hardware","name":"Hardware & Compute","url":"https://digestai.news/category/hardware"},"entities":{"companies":["NVIDIA","Anthropic","LMSYS Org","SemiAnalysis"],"models":["Opus 4.8","Fable"],"people":[]},"firstPublishedAt":"2026-09-21T18:14:15Z","updatedAt":"2026-09-21T18:14:15Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"SemiAnalysis","title":"Computation and Data Movement for Inference","url":"https://newsletter.semianalysis.com/p/computation-and-data-movement-for","publishedAt":"2026-09-21T18:14:15Z","type":"newsletter","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":{"title":"MoE Drives New AI Hardware Standards","url":"https://digestai.news/thread/nvidia-rubin-ultra-shifts-to-8-hi-hbm-4-hi-stacks-emerge-as-inference-cost","storyCount":2},"cite":{"text":"Digest AI, \"Mixture of experts reshapes inference compute and data movement pipelines\", 21 September 2026, https://digestai.news/story/mixture-of-experts-reshapes-inference-compute-and-data-movement-pipeli","publisher":"Digest AI","title":"Mixture of experts reshapes inference compute and data movement pipelines","datePublished":"2026-09-21T18:14:15Z","url":"https://digestai.news/story/mixture-of-experts-reshapes-inference-compute-and-data-movement-pipeli"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}