Developing story · 2 episodes · 13 Sept to 21 Sept
MoE Drives New AI Hardware Standards
The saga tracks how the rise of mixture-of-experts models is driving a fundamental shift in AI hardware architecture. It currently stands at a point where Nvidia's Rubin Ultra is optimizing for inference costs by adopting 8-hi HBM, while 4-hi stacks emerge as the new standard for efficient data movement.
-
Mixture of experts reshapes inference compute and data movement pipelines
Mixture‑of‑Experts (MoE) models have altered the architecture of inference services beyond simply adding parameters. Inference now runs in a cluster managed by orchestration layers such as NVIDIA…
1 source -
Nvidia Rubin Ultra shifts to 8-hi HBM; 4-hi stacks emerge as inference cost optimum
SemiAnalysis reports a structural shift in AI hardware design, marking the end of the trend toward ever-increasing High Bandwidth Memory (HBM) density per chip. Nvidia’s upcoming Rubin Ultra…
1 source
Who and what
NvidiaSemiAnalysisCerebrasGroqAnthropicLMSYS OrgRubin UltraBlackwell UltraKimi K3Llama 3.1 405BOpus 4.8Fable