{"version":1,"type":"story","url":"https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse","json":"https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse.json","markdown":"https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse.md","slug":"deepseek-releases-deepinfer-open-source-inference-framework-for-deepse","headline":"DeepSeek releases DeepInfer open-source inference framework for DeepSeek models","summary":"DeepSeek announced the open‑source release of DeepInfer, a high‑performance inference framework designed to serve its large‑scale production services. The framework supports the DeepSeek‑V4.1‑Flash model and provides offline inference with features such as optimized kernels, parallelism options, asynchronous scheduling, deterministic sampling, and structured output constraints. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs and requires Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend).\n\nDeepInfer includes custom kernels like FlashMLA and DeepGEMM, a draft‑token verification system (DSpark), and a batch‑invariant inference mode that keeps results identical across batch sizes on NVIDIA hardware. Benchmarks were measured on four SM100 GPUs for prefill and sixty‑four GPUs for decode, each within a single scale‑up domain. The default checkpoint, deepseek‑ai/DeepSeek‑V4.1‑Flash‑1017, is fetched from the Hugging Face cache on first run. The project is released under the Apache License 2.0 and builds on vLLM with inspiration from SGLang and TensorRT‑LLM.","keyPoints":["DeepInfer supports the DeepSeek‑V4.1‑Flash model and runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs","Features include optimized kernels, asynchronous batch scheduling, deterministic sampling, and structured output constraints","Benchmarks were run on 4 SM100 GPUs for prefill and 64 GPUs for decode, all under a single scale‑up domain"],"whyItMatters":"An open‑source, production‑grade inference engine lets developers and enterprises deploy large language models more efficiently, reducing reliance on proprietary stacks.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":["DeepSeek","DeepSeek-AI"],"models":["DeepSeek-V4.1-Flash"],"people":[]},"firstPublishedAt":"2026-10-16T00:00:00Z","updatedAt":"2026-10-16T00:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"github.com","title":"Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models!","url":"https://github.com/deepseek-ai/DeepInfer","publishedAt":"2026-10-16T00:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[{"site":"Reddit","url":"https://www.reddit.com/r/LocalLLaMA/comments/1xanm55/deepseek_released_deepinfer_open_source/","points":null}],"thread":{"title":"DeepSeek Advances AI Infrastructure With New Tools","url":"https://digestai.news/thread/deepseek-plans-to-use-huawei-ascend-950dt-for-pretraining-by-end-of-2026","storyCount":2},"cite":{"text":"Digest AI, \"DeepSeek releases DeepInfer open-source inference framework for DeepSeek models\", 16 October 2026, https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse","publisher":"Digest AI","title":"DeepSeek releases DeepInfer open-source inference framework for DeepSeek models","datePublished":"2026-10-16T00:00:00Z","url":"https://digestai.news/story/deepseek-releases-deepinfer-open-source-inference-framework-for-deepse"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}