DeepInfer is the inference framework that DeepSeek uses for its own large-scale production services, released as open source on GitHub in October 2026. This first release focuses on offline inference and currently supports one model family, DeepSeek-V4.1-Flash; the default checkpoint is DeepSeek-V4.1-Flash-1017 from Hugging Face.
The README lists optimized kernels and runtime (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels), attention data parallelism with balanced context parallelism, MoE expert parallelism with EPLB load balancing, DSpark draft-token decoding with confidence-scheduled verification, bounded sliding-window replay to cut prefill work, an asynchronous zero-overhead batch scheduler, batch-invariant inference and deterministic sampling (NVIDIA only), and structured outputs constrained by regex, JSON schema or EBNF grammar.
It is not a drop-in serving stack. DeepSeek states that a complete inference service also needs a KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas, which are not part of this release. Hardware support is limited to NVIDIA SM100 GPUs (CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2) on Linux with Python 3.12, and the project is built from source.
DeepSeek credits vLLM as the original base of DeepInfer, with inspiration from SGLang and TensorRT-LLM. Teams already serving DeepSeek models on supported hardware can use it as a reference for DeepSeek's production optimizations; for general multi-model serving, vLLM and SGLang remain the broader options. DeepInfer is free under the Apache License 2.0.
