DeepSeek open-sources DeepInfer, an inference engine for its V4.1 Flash model, but leaves out the KV cache
- DeepSeek published DeepInfer on GitHub, and its README says a complete inference service still needs a KV cache system and prefill/decode disaggregation that are not included in this release.
- The engine is the basis for serving DeepSeek V4.1 Flash, and a Zhihu post quoted on Hacker News claims one cluster handles 155 trillion tokens per day at about $38 per card per hour at off-peak pricing.
- A Hacker News commenter says the code is built on vLLM, which would make its novelty narrower than the release suggests, while others point out that much of the design already appears in DeepSeek's papers.
- The repo is young with three commits, and the core folders include csrc, deepinfer, and deep_cluster, with a Q&A doc updated an hour before the article was captured.
Hacker News opinions
Pretty wild if the $38 per card per hour number holds. Even with a DC full of GPUs the amortized cost is way below that, so inference clearly makes money, bubble talk aside.
Deepseeks strategy looks like each project has to break even in 10 months. At $38/hr that's about $320K per card, so the margin math is not subtle.
Can someone explain how an open source inference engine removes the moat for US labs? I don't know enough about LLMs to see it.
It's a project based on vLLM, so I'm not sure why people are calling it a total game changer.
I don't see the big deal. vLLM and SGLang already do this kind of serving, and a lot of the novel bits are already in DeepSeek's published papers. People will port the good parts over and everyone benefits.
The KV cache system is notably missing from this release. The README says a complete service needs it, plus prefill/decode disaggregation and load balancing, and none of that is included.
Data is the real prize IMO. Chinese labs are GPU constrained after the Nvidia and ASML embargoes, so they release open weights and let US and EU third parties eat the inference cost.
If open source inference engines spread across models and hardware, I'd expect pricing pressure pushing revenue toward the base card cost. That's bad for compute providers.