Nvidia Rubin Ultra shifts to 8-hi HBM; 4-hi stacks emerge as inference cost optimum

The analysis argues that 4-hi HBM stacks offer the best cost-per-bandwidth ratio, making them ideal for inference where bandwidth constraints dominate. While pre-training requires high capacity to store weights and gradients, the industry’s focus has shifted toward inference and reinforcement learning, which are bandwidth-sensitive. As scale-up domains expand to racks like NVL576, aggregate system memory now far exceeds the requirements for model weights, rendering excessive per-chip HBM capacity 'stranded' and inefficient.

Consequently, hardware teams at major AI labs are moving toward 4-hi configurations for next-generation ASICs. This approach maximizes tokens per HBM wafer, addressing supply chain bottlenecks while optimizing total cost of ownership. The shift prioritizes efficient bandwidth utilization over maximizing per-die capacity, aligning hardware design with the economic realities of serving large-scale inference traffic.