Engram offloading boosts DeepSeek-V4.1-Flash performance on B300 by up to 1.6x
The SemiAnalysis report describes Engram, a token‑embedding extension that stores recurring patterns in a lookup table. By moving the Engram table—about 24 rows per layer, roughly 12.4 KiB per token position (3.1 KiB per GPU when split across four GPUs)—out of high‑bandwidth memory (HBM) into host DRAM, the model can keep HBM free for weights and KV cache. DeepSeek‑V4.1‑Flash’s Engram table uses…
Key points
- DeepSeek-V4.1-Flash’s Engram table occupies roughly 189 GiB; the report tests offloading it to DRAM and SSD across six NVIDIA GPUs and AMD MI355X.
- Offloading Engram to DRAM on B300 improves the pareto curve by up to 1.6x, reducing HBM demand for the model.
- DRAM serving yields 121 million total tokens per dollar versus 52 million for SSD; SSD offloading shows no cost advantage.
The report notes that AMD’s MI355X remains 2‑4x worse performance per dollar than B200, and performance is up to 14.8x worse per dollar than H200 and up to 42x worse per dollar than B200/B300. DRAM serving delivers about 121 million total tokens per dollar versus 52 million for SSD, with near 125 tokens / s / user. The authors claim the benchmark has been widely reproduced and supported by major cloud providers such as Google Cloud, Microsoft Azure, Oracle and Meta, as well as ML frameworks like vLLM, LMCache and HuggingFace.
Model page: DeepSeek-V4.1-Flash →
The story so far
3 episodes →- Engram offloading boosts DeepSeek-V4.1-Flash performance on B300 by up to 1.6xthis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Hardware & Compute
All →- Apple reportedly developing AI-powered smart glasses without built-in display · 1 src
- Apple plans 2029 AI server launch using M8 Ultra and Nvidia NVLink tech · 7 src
- OpenAI and Anthropic are reportedly exploring 20-30 MW AI compute clusters in the UK and Nordics · 4 src
- Huawei Announces New AI Chips to Compete with Nvidia · 5 src
- Apple iPhone 18 Pro reportedly tested with variable aperture camera and Siri AI · 1 src
Comments
via GitHub Discussions