ings
DeepSeek has open-sourced DeepInfer, the inference framework behind its large-scale production services, alongside deepseek-recipe, a library collection that connects model inputs and outputs with common API formats. Together, the releases give developers access to more of the software surrounding its models. Sources: DeepInfer and deepseek-recipe.
Making Inference More Efficient
DeepInfer supports DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Its features include distributed execution, speculative decoding, asynchronous scheduling and structured outputs.
The optimizations address different sources of wasted computation. Speculative decoding proposes tokens for verification, while confidence estimates help schedule that verification. Asynchronous scheduling overlaps CPU work with accelerator execution. Bounded replay reduces repeated processing by revisiting only a limited suffix of the input. Source: DeepInfer documentation.
Connecting Models to Applications
deepseek-recipe handles the layer between applications and inference engines. Written in Rust with Python bindings, it converts Messages, Chat Completions and Responses requests into a shared conversation format, prepares DeepSeek prompts and processes model outputs into API responses.
It supports streaming, reasoning content, images and tool-call parsing. For developers, this can reduce the work involved in adapting applications to a different model backend.
However, the library does not execute the model or run tools. Developers must still supply inference, tool execution and HTTP transport. Matching an API format alone does not establish identical behavior across deployments. Source: deepseek-recipe documentation.
The Engine Is Available. The Full Service Still Takes Work.
The DeepInfer release provides offline inference, but a complete service still requires components absent from the release: a KV cache system, separated vision/prefill/decode deployment, and load balancing across replicas. DeepInfer uses the Apache 2.0 license; deepseek-recipe uses MIT. Sources: DeepInfer and deepseek-recipe.
For teams building their own deployments, the value is access to concrete engineering choices they can inspect and test. Downloading the engine is one step; running it reliably and economically is the next.
