DeepSeek Open-Sources DeepInfer, the Engine Behind V4.1 Flash -- Its Deployment Notes Say One Cluster Serves 155 Trillion Tokens a Day, and the Engine Targets Both NVIDIA and Huawei

On October 20, 2026, DeepSeek made its first commit to deepseek-ai/DeepInfer, which the README describes as "the high-performance inference framework that powers DeepSeek's large-scale production services." By the next morning it had more than 13,000 GitHub stars, and its Hacker News post was the day's top story with 2,145 points and 838 comments. The release supports one model, DeepSeek-V4.1-Flash, and two hardware targets: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.

What is in the box, and what is not

The open code covers offline inference. It includes DeepSeek's kernel libraries (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect) plus custom TileLang kernels; attention data parallelism with MoE expert parallelism and expert load balancing; DSpark draft tokens with "confidence-scheduled verification"; structured outputs constrained by regex, JSON schema or EBNF; and, on NVIDIA only, batch-invariant inference and deterministic sampling, which keep "model results identical across different batch compositions." The acknowledgements say DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM." The README is explicit about what is missing for a full service: "A KV cache system", prefill/decode disaggregation, and "Load balancing across data-parallel replicas". The Q&A explains that these parts are "tightly coupled to our internal infrastructure." DeepSeek also released deepseek-recipe, a tokenizer layer matched to its official API that segments on special tokens, which it says "prevents special-token injection".

The numbers in the deployment notes

• Scale: "A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing)." By our arithmetic, that is about 1.8 billion tokens a second.
• Shape: prefill on "ordinary 8-GPU machines" at 2-8-way expert parallelism, and decode on 64-128-way "Wide-EP", so the smallest efficient unit is 64 prefill plus 64 decode GPUs. Load balancing reaches "roughly 80% of a perfectly balanced offline benchmark."
• Cache: about 100 PiB of SSD on DeepSeek's 3FS file system, with 10+2 erasure coding. KV cache is kept for 3 days for agent traffic and 7 for chatbot traffic, because chat has a heavier long tail of repeated hits; a 3-day limit would cost chat 0.5-1% of its hit rate.
• Precision: MXFP4 for routed-expert weights and an FP4 format for the main KV cache.

The engineering detail is unusually frank. Prefill and decode can run in different data centres, but "the fiber running between data centers is easily severed by construction work", so DeepSeek uses adjacent ones.

What remains uncertain. The published charts are DeepSeek's own runs, on 4 SM100 GPUs for prefill and 64 for decode, and none are for Ascend. The Q&A gives no cluster size, so the throughput and revenue figures cannot be turned into a margin, and "valley pricing" means off-peak rates. With the cache and routing layers withheld, nobody outside DeepSeek can reproduce production performance yet. The release also lands a day after a US Homeland Security order, covered here on October 20, that reaches products "developed using DeepSeek implementations", so US government suppliers will need to know whether their stack contains this code.

Analysis: The code matters less than the operating manual. Several patterns carry over to any MoE serving stack: a shared KV store instead of prefix-affinity routing, cache lifetimes set per traffic class, prefill chunk sizes solved from a fitted cost model rather than fixed, and a one-minute grace period that moves in-flight requests and their cache off a draining decode engine. Batch invariance is worth copying too. If outputs do not depend on what else is in the batch, evaluations become reproducible and agent bugs can be replayed. Putting Ascend next to NVIDIA as a first-class target is a hardware signal in its own right.