# Engram offloading boosts DeepSeek-V4.1-Flash performance on B300 by up to 1.6x

Digest AI · Hardware & Compute · published 2026-09-18T14:34:20Z

Canonical: https://digestai.news/story/engram-offloading-boosts-deepseek-v4-1-flash-performance-on-b300-by-up

## Summary

The SemiAnalysis report describes Engram, a token‑embedding extension that stores recurring patterns in a lookup table. By moving the Engram table—about 24 rows per layer, roughly 12.4 KiB per token position (3.1 KiB per GPU when split across four GPUs)—out of high‑bandwidth memory (HBM) into host DRAM, the model can keep HBM free for weights and KV cache. DeepSeek‑V4.1‑Flash’s Engram table uses roughly 189 GiB of memory; the authors tested a memory‑mapped file on SSD and a DRAM offload across six NVIDIA GPU SKUs (H100, H200, B200, B300, GB200, GB300) and AMD’s MI355X. Offloading on B300 improves the pareto curve by up to 1.6x, allowing a switch from TP4 to TP2 and reducing HBM demand.

The report notes that AMD’s MI355X remains 2‑4x worse performance per dollar than B200, and performance is up to 14.8x worse per dollar than H200 and up to 42x worse per dollar than B200/B300. DRAM serving delivers about 121 million total tokens per dollar versus 52 million for SSD, with near 125 tokens / s / user. The authors claim the benchmark has been widely reproduced and supported by major cloud providers such as Google Cloud, Microsoft Azure, Oracle and Meta, as well as ML frameworks like vLLM, LMCache and HuggingFace.

## Key points

- DeepSeek-V4.1-Flash’s Engram table occupies roughly 189 GiB; the report tests offloading it to DRAM and SSD across six NVIDIA GPUs and AMD MI355X.
- Offloading Engram to DRAM on B300 improves the pareto curve by up to 1.6x, reducing HBM demand for the model.
- DRAM serving yields 121 million total tokens per dollar versus 52 million for SSD; SSD offloading shows no cost advantage.

## Why it matters

Reducing HBM requirements via Engram offloading lets operators run larger models or higher batch sizes on existing GPUs, lowering inference cost and expanding deployment options.

## Sources

1. [Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading](https://newsletter.semianalysis.com/p/engrams-embedding-entendre-codesign) (SemiAnalysis, 2026-09-18)

Part of the developing story: [DeepSeek Launches Open Long Context Agent Model](https://digestai.news/thread/deepseek-v4-1-flash-reduces-ai-agent-memory-usage) (3 stories)

## Cite

Digest AI, "Engram offloading boosts DeepSeek-V4.1-Flash performance on B300 by up to 1.6x", 18 September 2026, https://digestai.news/story/engram-offloading-boosts-deepseek-v4-1-flash-performance-on-b300-by-up

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/engram-offloading-boosts-deepseek-v4-1-flash-performance-on-b300-by-up.json