DeepSeek releases DeepInfer open-source inference framework for DeepSeek models
DeepSeek announced the open‑source release of DeepInfer, a high‑performance inference framework designed to serve its large‑scale production services. The framework supports the DeepSeek‑V4.1‑Flash model and provides offline inference with features such as optimized kernels, parallelism options, asynchronous scheduling, deterministic sampling, and structured output constraints. It runs on NVIDIA…
Key points
- DeepInfer supports the DeepSeek‑V4.1‑Flash model and runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
- Features include optimized kernels, asynchronous batch scheduling, deterministic sampling, and structured output constraints
- Benchmarks were run on 4 SM100 GPUs for prefill and 64 GPUs for decode, all under a single scale‑up domain
DeepInfer includes custom kernels like FlashMLA and DeepGEMM, a draft‑token verification system (DSpark), and a batch‑invariant inference mode that keeps results identical across batch sizes on NVIDIA hardware. Benchmarks were measured on four SM100 GPUs for prefill and sixty‑four GPUs for decode, each within a single scale‑up domain. The default checkpoint, deepseek‑ai/DeepSeek‑V4.1‑Flash‑1017, is fetched from the Hugging Face cache on first run. The project is released under the Apache License 2.0 and builds on vLLM with inspiration from SGLang and TensorRT‑LLM.
The story so far
2 episodes →- DeepSeek releases DeepInfer open-source inference framework for DeepSeek modelsthis story
Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models!
github.com · 16 October 2026Loading the full article…
This text was published by github.com. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Reddit discussionreddit.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Google Gemini 4 Argon agents beat Minecraft's Ender Dragon without human intervention · 4 src
- Egen becomes launch partner for Google Cloud Gemini for government services · 1 src
- Exa claims Exa 3.0 search API leads Perplexity, Brave and Parallel on eight benchmarks · 1 src
- U.S. Army develops echotribble program to deploy autonomous AI agents for cyber attacks · 1 src
- Ready Server adds free ReadyServer LLM API to every VPS plan · 1 src
Comments
via GitHub Discussions