DeepInfer: DeepSeek open-sources its V4.1 Flash inference engine

DeepSeek has open-sourced the engine it says runs V4.1 Flash in production. It is called DeepInfer, it is Apache-2.0, and it runs on both NVIDIA and Huawei hardware. But the repository ships the inference core, not the whole serving system, and the numbers about DeepSeek’s own cluster are its own claims. We read the README and the deployment Q&A in the repository through the GitHub API. This article covers what you can run today, what DeepSeek kept back, and which claims are not independently checkable.

What is DeepInfer?

A large-scale inference framework for DeepSeek models, released on GitHub under Apache-2.0. The README calls it “the high-performance inference framework that powers DeepSeek’s large-scale production services” and says this release provides offline inference with the core features needed for serving at scale. GitHub shows the repository was created on October 16 and the first public commit landed on October 20 (04:09 UTC). It is a very new release: at our reading the commit history had three commits, no tagged releases and no contributor list. It was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM, per its acknowledgments.

What can you run today?

Offline inference for DeepSeek-V4.1-Flash, on two kinds of accelerators. The supported model is V4.1 Flash, with the default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 downloaded from Hugging Face on first run. Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Building needs Linux, GCC 13 or later, Python 3.12, and CUDA 13 on NVIDIA or CANN 9.2 on Ascend. The listed features:

• Kernels and runtime: optimized kernels, a model and sampler runtime, with FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels as integrations.
• Parallelism: attention data parallelism with balanced context parallelism, and MoE expert parallelism with expert-load balancing (EPLB).
• Decoding: DSpark and confidence-scheduled verification, with a default maximum draft length of 5, and “SWA bounded replay” to cut decoder prefill work.
• Scheduling: asynchronous, enabled by default.
• Reproducibility: batch-invariant inference and deterministic sampling, both NVIDIA only.
• Structured outputs: constrain generation to a regex, JSON schema or EBNF grammar.

DeepSeek also says it is open-sourcing a companion project, deepseek-recipe, a production token encoding and decoding component. We did not open it.

What is not included?

The parts that turn an engine into a service. The README lists three things “required for a complete inference service” that are not in this release: a KV cache system, prefill and decode (and vision encoder) disaggregation, and load balancing across data-parallel replicas for both prefill and decode. The Q&A adds that “most required engine functionality is already implemented” for disaggregation, and that the KV cache management and load-balancing components are “tightly coupled to our internal infrastructure” and not yet fully open. Model weights are separate and are fetched on first run.

How does DeepSeek say its system works?

As a disaggregated design with a wide expert-parallel decode stage. The Q&A states three goals: support accelerators from multiple vendors, organize prefill and decode around commodity hardware, and retain KV cache for more than 72 hours. It describes:

• Prefill on ordinary 8-GPU machines with EP2 to EP8, and decode on 64 to 128 GPU supernodes with EP64 to EP128. It says the smallest efficient deployment unit is 64 prefill GPUs plus 64 decode GPUs.
• A KV cache store on a dedicated 3FS storage cluster of about 100 PiB of SSD, with a three-day time-to-live for agent traffic and seven days for chatbot traffic, and a stated cost of 0.5% to 1% of cache hit rate for chatbot traffic at around three days.
• Load balancing that it says reaches roughly 80% of a perfectly balanced offline benchmark.
• A multimodal path with a single-GPU vision encoder per instance.

These are DeepSeek’s descriptions of its own system. Most of the parts they describe are the ones not in the release.

What are the 155 trillion tokens a day?

DeepSeek’s claim about its own cluster. The Q&A says a single cluster currently serves “155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing).” It does not give the cluster size, the token mix, or what valley pricing is. We cannot check either figure. The repository’s performance section shows Pareto charts for prefill (measured on 4 SM100 GPUs) and decode (64 GPUs) and a sweep script to reproduce them, but no table of numbers, and we could not read exact values from the charts.

What is still unknown?

• Whether the open release matches production. The README says the framework powers production, but the missing components are what production serving depends on.
• How fast it is. The charts are the only performance evidence, and nothing here is independent.
• Which other models it will support. Only V4.1 Flash is listed.
• Whether DeepSeek will open the KV cache and load-balancing code. The Q&A says “not yet fully open-sourced” and gives no date.
• How the model name and checkpoint relate. The README lists “DeepSeek-V4.1-Flash” with a “-1017” default checkpoint and does not explain the naming.

What to keep in mind

These are our observations:

• This is an engine, not a serving stack. You can run the model with DeepSeek’s kernels. You cannot rebuild its described production cluster from this repository alone.
• The Ascend support is part of the story. A Chinese lab shipping inference code for Huawei Ascend 950 and NVIDIA together is notable, and it lands in the same week the US Department of Homeland Security made public an order removing DeepSeek products from federal civilian IT.
• Check the license against your own use. The code is Apache-2.0. The model weights are governed by their own license, which we did not review here.
• It builds on our other DeepSeek coverage. See the V4.1-Flash-1017 release and the DeepSeek harness.

Editorial review: October 20, 2026 (AI). Prepared from the DeepInfer repository’s README and its deployment Q&A, both read through the GitHub API, plus the repository’s creation and commit dates from the GitHub API. Some long passages in the Q&A were truncated in our reading. We did not build or run DeepInfer, open the deepseek-recipe project, read the Zhihu post the Q&A links, or check any figure. We did not compare the open release with DeepSeek’s hosted service.