# DeepSeek releases DeepInfer open-source inference framework for DeepSeek models

Digest AI · Agents & Tools · published 2026-10-16T00:00:00Z

Canonical: https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse

## Summary

DeepSeek announced the open‑source release of DeepInfer, a high‑performance inference framework designed to serve its large‑scale production services. The framework supports the DeepSeek‑V4.1‑Flash model and provides offline inference with features such as optimized kernels, parallelism options, asynchronous scheduling, deterministic sampling, and structured output constraints. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs and requires Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend).

DeepInfer includes custom kernels like FlashMLA and DeepGEMM, a draft‑token verification system (DSpark), and a batch‑invariant inference mode that keeps results identical across batch sizes on NVIDIA hardware. Benchmarks were measured on four SM100 GPUs for prefill and sixty‑four GPUs for decode, each within a single scale‑up domain. The default checkpoint, deepseek‑ai/DeepSeek‑V4.1‑Flash‑1017, is fetched from the Hugging Face cache on first run. The project is released under the Apache License 2.0 and builds on vLLM with inspiration from SGLang and TensorRT‑LLM.

## Key points

- DeepInfer supports the DeepSeek‑V4.1‑Flash model and runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
- Features include optimized kernels, asynchronous batch scheduling, deterministic sampling, and structured output constraints
- Benchmarks were run on 4 SM100 GPUs for prefill and 64 GPUs for decode, all under a single scale‑up domain

## Why it matters

An open‑source, production‑grade inference engine lets developers and enterprises deploy large language models more efficiently, reducing reliance on proprietary stacks.

## Sources

1. [Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models!](https://github.com/deepseek-ai/DeepInfer) (github.com, 2026-10-16, primary source)

Part of the developing story: [DeepSeek Advances AI Infrastructure With New Tools](https://digestai.news/thread/deepseek-plans-to-use-huawei-ascend-950dt-for-pretraining-by-end-of-2026) (2 stories)

## Cite

Digest AI, "DeepSeek releases DeepInfer open-source inference framework for DeepSeek models", 16 October 2026, https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse.json