DeepSeek Open-Sources DeepInfer, the Inference Framework Behind DeepSeek-V4.1-Flash

DeepSeek has released DeepInfer, the inference engine it says runs its own production services and powers DeepSeek-V4.1-Flash, under the Apache 2.0 license. Alongside it comes deepseek-recipe, an MIT-licensed layer for encoding and decoding tokens. Frontier labs almost never publish the code that actually serves their traffic, so this is a rare look behind the counter.

The numbers DeepSeek attaches are striking, and all self-reported: a single cluster serving 155 trillion tokens a day, decode throughput above 25,000 output tokens per second per GPU on a 64-GPU setup at 131,072-token context, and revenue of about US$38 per GPU-hour at its off-peak pricing. If those figures hold up in outside hands, they explain a lot about how DeepSeek keeps undercutting rivals on price.

We think this is a genuinely useful release, and we want to be precise about why. It is not a drop-in replacement for vLLM or SGLang. It supports exactly one model, runs on two hardware targets — NVIDIA's Blackwell-class SM100 parts and Huawei's Ascend 950 — and handles offline inference only. What it offers is a worked example of how a lab that lives and dies on serving cost actually squeezes its hardware.

The most interesting idea for us is DSpark, a twist on speculative decoding. A small draft model proposes five tokens, a confidence head guesses which of them will survive, and the system only verifies the promising prefix. DeepSeek says that cuts verified draft tokens by roughly 30% while accepted length barely moves. The gain is real but modest — a few percent of throughput at moderate batch sizes, and slightly slower at small ones. That honesty in the published curves is worth crediting. The bounded replay trick for sliding-window attention, which recomputes only a fixed tail of tokens on a cache hit, is the kind of engineering detail that other serving teams will borrow quickly.

Now the catch. The pieces that make a production service a service — the KV cache system, prefill/decode disaggregation, load balancing across replicas — are not included, and DeepSeek says they are tied to internal infrastructure. The smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs. And the internal attention kernel is still a bit faster than the public one, though DeepSeek puts the end-to-end gap at 5% or less. So this is open in the sense of a published blueprint, not open in the sense that a mid-size company can stand up DeepSeek's economics next week.

The hardware story deserves its own attention. First-class support for Huawei's Ascend 950 next to Blackwell is a statement. Every serving stack that runs well on non-NVIDIA silicon chips away at the idea that frontier-grade AI requires one supplier, and it gives Chinese labs a more credible domestic path under export pressure. Whether one welcomes that or worries about it, it changes the competitive map.

Where does this lead if it becomes normal? We would like a world where labs publish their serving engines, because inference cost decides who gets access to capable models and at what price. The downside is concentration of a different kind: the techniques reward operators who can afford hundreds of top-tier accelerators, so the open code may widen the gap between hyperscale serving and everyone else even as it educates them. I'm watching whether the vLLM and SGLang communities absorb DSpark-style scheduling, because that is how this release reaches ordinary deployments.

DeepInfer was originally built largely on vLLM, with ideas taken from SGLang and NVIDIA's TensorRT-LLM, and sits on top of DeepSeek's previously released kernel libraries such as FlashMLA, DeepGEMM and DeepEP. Builds require Linux, Python 3.12, GCC 13 or later, and either CUDA 13 or Huawei's CANN 9.2. Structured output supports regex, JSON schema and EBNF grammars; deterministic sampling is available on NVIDIA only.