vLLM 0.28 adds Kimi-K3 and DeepSeek V4 inference work, tiered KV cache offloading
- vLLM 0.28.0 contains 584 commits from 270 contributors, including 76 first-time contributors.
- For Kimi-K3, vLLM adds Decode Context Parallelism, fused FlashKDA decode and prefill kernels, ROCm support with the V2 model runner, and an optional shared-expert sharding mode that reports saving about 17 GiB per GPU.
- The release adds end-to-end sparse MLA support for DeepSeek V4 plain decoding, MTP, and DSpark speculative decoding, plus ROCm enablement on gfx11 and gfx950 GPUs.
- Speculative decoding gains DFlash2 with local convolution and candidate selection, DSpark confidence-scheduled verification, and automatic async scheduling for draft models.
- Tiered KV cache offloading now supports disk storage, third-party secondary-tier managers via module_path, partial secondary-tier loads, metrics, and a parallelism-agnostic canonical CPU layout. The V2 runner also gains E/P/D disaggregation and weight offloading.
Hacker News opinions
I love vLLM, but recent releases have been frustratingly buggy for me. DeepSeek-V4-Flash was broken on a B300 in 0.26, then 0.27 stopped needing my patches but began producing repeated-token garbage; Gemma-4 on an RTX 6000 also hangs until I kill and restart it. High concurrency on a friend's 4x RTX 6000 setup produces malformed output too.
I've had DeepSeek 4 Flash instability too, but it stabilized for me on 2x H200 with a commit a few days before 0.28. This release should be better for that case.
My team runs open models for company developers, mostly on H200s. If you deploy bleeding-edge vLLM or newly released models, expect some patching; we use A/B deployments and end-to-end tests with opencode, codex, and claude to catch failures.
I've seen the same random gibberish where the model seems unable to finish. In my case, a patch stored KV-cache indices in variable types that were too narrow, so they wrapped around.
I was hoping for a real fix to the reasoning_content mess. This release appears to contain only a documentation change for it.
I load-tested vLLM and managed to crash it.
vLLM is still behind llama.cpp on sampler support. I want top-n-sigma, DRY, and XTC support.
If you need those samplers, submit a pull request.