DeepInfer explained: how DeepSeek's open-source inference engine serves V4.1 Flash, and what it takes to run it

On 20 October 2026 DeepSeek open-sourced DeepInfer, the inference framework behind its production DeepSeek-V4.1-Flash service, under the Apache 2.0 licence. It reached the top of Hacker News with more than 1,800 points within hours. It is not a drop-in replacement for vLLM: it serves one model, needs Blackwell-class NVIDIA GPUs or Huawei Ascend 950 NPUs, and ships offline inference only, without the KV cache store and load balancers that turn an engine into a service.

What it does ship is unusually candid: the kernels, parallelism layout, speculative decoding scheduler and cache tricks DeepSeek uses in production, plus a deployment Q&A with real operating numbers. This article is for engineers who run or plan LLM inference. You will see how the system is laid out, which ideas you can borrow even if you never run DeepInfer, what hardware it needs, and how to build it and run your first batch.

What is DeepInfer?

DeepInfer is DeepSeek's in-house LLM inference engine, released as "offline inference with the core features needed for serving at scale", in the words of its README. It combines DeepSeek's earlier kernel libraries, FlashMLA, DeepGEMM and DeepEP, with TileKernels, DeepSelect and custom TileLang kernels, under a scheduler and model runtime that began life as a vLLM fork.

The scale behind it is the reason people are paying attention. The deployment Q&A says a single cluster serves 155 trillion tokens a day at an overall revenue of US$38 per GPU-hour at off-peak ("valley") pricing. In April 2025, as The New Stack recalls, DeepSeek had declined to release its engine because it was too tightly coupled to internal infrastructure. This release keeps exactly the coupled parts back.

What hardware does DeepInfer need?

Blackwell-generation data centre GPUs (NVIDIA SM100) or Huawei Ascend 950 NPUs. Hopper (H100, H200) and older architectures are not listed, as SiliconANGLE also notes, and nothing in the build suggests a fallback path. The NVIDIA build pins torch==2.13.0+cu132 in its requirements and patches NCCL to 2.32.3 during build.sh.

The sizes are not workstation sizes either. DeepSeek's published benchmarks used 4 GPUs for prefill and 64 for decode, each inside one scale-up domain. The Q&A says a deployment with one prefill and one decode instance works, but because decode is recommended at expert parallelism of 64 or more and the prefill-to-decode GPU ratio is roughly 1:1, "the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode".

How DeepSeek serves V4.1 Flash: the architecture

The production system splits each request across three kinds of instance, vision encoder, prefill and decode ("E/P/D disaggregation"), with a dedicated KV cache storage cluster beside them. Attention runs data-parallel and the mixture-of-experts layers run expert-parallel, with the number of DP replicas equal to the EP size. Tensor and context parallelism are not used in production at present, according to the Q&A.

Why decode runs on 64 to 128 GPUs at once

Wide expert parallelism cuts the memory traffic each GPU spends on MoE experts per step, which raises tokens per second. DeepSeek therefore runs prefill at the smallest EP that fits in HBM (EP2–8) and decode at the largest the GPUs allow (EP64–128), and publishes an EP8 versus EP64 decode chart in the Q&A.

Wide EP only pays off if experts are balanced, because the slowest GPU sets the step time. DeepInfer's EPLB keeps redundant copies of hot experts and reshuffles them over NVLink. For prefill it starts from 4 redundant experts per GPU, measures load over 100 steps and reassigns every 1,000 steps; decode uses 1 redundant expert, a 1,000-step window and reassignment every 10,000 steps. The measured ratio of slowest rank to mean was 1.045 to 1.074 for prefill and 1.090 for decode. Across replicas, DeepSeek says its balancers reach about 80% of a perfectly balanced offline benchmark: a useful reality check for any capacity plan built on benchmark numbers.

DSpark: speculative decoding that verifies only what is likely to be accepted

DSpark has a draft model propose 5 tokens per step, then a confidence head predicts how likely each draft token is to survive. Instead of verifying all 5, the scheduler verifies the prefix that maximises expected throughput on a profiled hardware curve. The DSpark paper reports 60–85% faster per-user generation than DeepSeek's previous MTP-1 baseline at matched throughput, measured on live traffic.

The engine-level gain of variable-length verification over always verifying 5 tokens is smaller and depends on load. DeepSeek's own table shows it cutting verified tokens per step by about 30% while accepted length barely moves (3.29 against 3.20), which helps only once the GPU is busy:

At 16 sequences per rank dynamic verification is 1.5% slower; at the production point of about 60 it is 5.4% faster, and at 96 it is 11.0% faster. If your traffic is light, expect nothing from this; if your GPUs are saturated, it is free throughput. The technique is worth copying in any engine with a confidence signal.

SWA bounded replay: constant-cost recompute on a cache hit

V4.1 Flash mixes paged block attention with sliding-window attention (SWA). The block KV cache goes to shared storage, but SWA state stays in host memory on the machine that produced it, so a prefix hit on another machine can find the block cache without the SWA state. Rebuilding it exactly would cost more as contexts grow. DeepInfer instead recomputes only the last 128 tokens and truncates attention to that segment:

Here s is where the replay starts and i is the query position, so the result is an approximate SWA cache at a cost independent of context length and layer count. In the decoder layers the same trick replaces running the last 2,560 tokens through 20 layers, which the Q&A says nearly halves total prefill computation. This is why the README asks for DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140: the 128-token window plus room for speculative backtracking.

The cache numbers behind the hits are large: about 100 PiB of SSD on 3FS with 10+2 erasure coding, a time-to-live of 3 days for agent traffic and 7 for chat, and roughly 14.4 TB of logical KV cache per online GPU. Cutting chat retention to 3 days cost 0.5–1% of hit rate, which tells you how much of chat traffic is the same long prefix read again and again.

How to build DeepInfer and run offline inference

There is no pip install. You build from source inside an empty Python 3.12 environment, with the target platform exported for both the build and every run.

1. 1

   Build for your accelerator

   git clone https://github.com/deepseek-ai/DeepInfer.git && cd DeepInferpython3.12 -m venv .venv && . .venv/bin/activate # 'ascend' for Ascend 950; keep it exported wherever you run DeepInferexport DEEPINFER_ACCELERATOR=nvidia./build.sh

2. 2

   Run the offline example

   # Downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 into the Hugging Face cachepython examples/offline_inference.py # Or use a local copy (no network access)DEEPINFER_HF_MODEL_DIR=/models/DeepSeek-V4.1-Flash-1017 python examples/offline_inference.py

   The script prints each sample's output with its accept_rate and decode_ppl, which the test suite parses against stored baselines, so a numerical change in your build shows up as a perplexity mismatch rather than silently.

3. 3

   Choose the parallel layout

   python examples/offline_inference.py --ep 8            # EP8, DP defaults to EP/CPpython examples/offline_inference.py --ep 2 --dp 8     # EP2 with 8 DP rankspython examples/offline_inference.py --ep 2 --cp 8     # context parallel

The engine API follows vLLM's shape: an LLM object, SamplingParams and generate(). Structured outputs are a sampling option rather than a server feature. This snippet, adapted from the repository's own check, constrains output to a JSON schema; regex and EBNF use the same so_args:

import jsonfrom deepinfer.sampling_params import SamplingParams json_schema = {    "type": "object",    "properties": {        "color": {"type": "string", "enum": ["red", "green", "blue"]},        "count": {"type": "integer", "enum": [1, 2, 3]},    },    "required": ["color", "count"],    "additionalProperties": False,}params = SamplingParams(    temperature=0,    max_tokens=128,    so_args={"schema_type": "json", "schema": json.dumps(json_schema)},)# llm is the engine built by create_llm() in examples/_config.pyoutputs = llm.generate(prompts=["Pick a colour and a count."], sampling_params=[params], use_tqdm=False)

For reproducible runs on NVIDIA, the README combines batch invariance with speculative decoding switched off and a fixed per-sequence seed: set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass seq_seed to SamplingParams. Turning DSpark on is DEEPINFER_ENABLE_MTP_VARLEN=1, with DEEPINFER_USE_NEXTN_DECODE capping the draft length at its default of 5.

deepseek-recipe: match the official API without running DeepInfer

The piece most teams can use today is deepseek-recipe, a Rust library with Python bindings that turns Chat Completions, Responses and Messages requests into DeepSeek V4 and V4.1 prompts and token IDs, and parses model output back into responses. DeepSeek says its logic is "fully aligned" with the official API, so a self-hosted model on any engine can behave like api.deepseek.com for tool calls, thinking mode and structured output.

# pip install deepseek-recipe   (Python 3.10+)from deepseek_recipe import ChatCompletionRequest, ConversionOptions, DeepseekV41Encoding request = ChatCompletionRequest({    "model": "DeepSeek-V4.1-Flash",    "messages": [{"role": "user", "content": "Hello"}],})converted = request.convert(ConversionOptions())rendered = DeepseekV41Encoding().render_conversation(converted.conversation)print(rendered.prompt)

It also closes a quiet security hole. The Q&A explains that strings such as <｜User｜> appearing in user input or tool results are encoded as ordinary text, not as control tokens, and that </think> and tool call markers are recognised only by token ID. A hand-rolled chat template that tokenises the whole prompt as one string does not give you that, and lets a pasted string impersonate a role. One more thing to know: its image fetcher connects to any address by default, so the README tells services that accept untrusted URLs to install a DestinationFilter.

Should you run DeepInfer or stay on vLLM or SGLang?

For most teams, stay where you are and borrow the ideas. DeepInfer makes sense if you serve DeepSeek-V4.1-Flash at high volume on SM100 or Ascend 950 hardware and can build your own KV cache tier and routing. If you need Hopper support, several models, or an OpenAI-compatible server out of the box, a general engine is still the practical choice, and DeepSeek says its internal FlashMLA is only 5–10% faster at the kernel level and at most 5% end to end than what it has published.

What to take from DeepInfer

• To do: If you run DeepSeek models on another engine, adopt deepseek-recipe now for API-compatible prompts and safe special-token handling.
• To do: Measure load imbalance (slowest rank divided by mean) before scaling expert parallelism; DeepSeek runs at 1.05–1.09.
• To do: Budget for about 80% of offline benchmark throughput once real routing is in place.
• To do: Treat confidence-scheduled verification as a high-load optimisation; it costs a little below about 16 sequences per rank.
• To do: Plan KV cache storage as its own tier with a TTL by traffic type, not as host memory on the inference nodes.
• To do: Only try DeepInfer itself on SM100 or Ascend 950, and expect to build the serving layer yourself.

Sources and further reading

• DeepInfer on GitHub, its deployment Q&A, examples and evaluation guide
• deepseek-recipe on GitHub
• DeepSeek-V4.1-Flash-1017 model card
• DSpark: Confidence-Scheduled Speculative Decoding (arXiv 2607.05147)
• DeepSeek's Zhihu post on the V4.1 Flash inference system (Chinese)
• Coverage: The New Stack, SiliconANGLE, and the Hacker News discussion