vLLM 0.31 adds constrained decoding, letting Ubicloud drop its fork and hit 100% tool-call schema accuracy
- vLLM 0.31 added constrained decoding, which blocks invalid tokens during sampling, so Ubicloud stopped carrying its own vLLM fork and still gets 100% tool-call schema accuracy with the newer Kimi and DeepSeek models.
- Ubicloud kept the fork because bad tool calls compound: a malformed call stays in the context and the model builds its next turn on the nonsense.
- An HN commenter read the example stats as a 23/25 tool-call success rate and calculated that backing up and retrying costs only +9% tokens on the call attempt, so the missing piece is a comparison of overall model quality with and without the constraint.
- Moonshot AI serves Kimi with constrained decoding on by default and OpenAI's Responses API turns it on by default, but Ubicloud notes published work shows it can hurt quality depending on the model.
- llama.cpp has shipped grammar-constrained decoding for years and transformers.js added it last month, so vLLM arrived late to a feature other serving engines treated as standard.
Hacker News opinions
I'm one of the authors of this post. Getting the LLM, the serving engine and the constrained decoding implementation to line up so you get the output you actually intended takes a lot of careful engineering and testing. Happy to answer questions.
Where does that <||> syntax even come from? It's odd that they didn't reuse existing markup, and there's almost nothing about it online.
So the model isn't really making the tool call, the engine just filters the sampler down to valid moves? Could you use that sampler massaging elsewhere to steer output while the weights stay the same?
Does this survive inference optimizations like DFlash and MTP, or does it slow things down to the point where retrying is just cheaper?
Hugging Face transformers.js got constrained decoding last month too, for in-browser structured output and tool calling.
Can the same mechanism force syntactically correct code?
Yes, all you need is a formal grammar for the output syntax. llama.cpp documents how grammar-constrained decoding works and ships a C example, and I've run text-to-SQL experiments with it. The hard part is wiring it into the inference engine, not impossible, just very hard.
Is that even a real problem though? If you're consistently getting syntax errors out of frontier models plus Codex or Claude Code, you've got bigger issues.
Constrained decoding is the same approach we used to turn an AR model into the /v1/systemone endpoint in vLLM. Tool calling is messy because nothing says a model's tool set is fixed for the life of a run, and the industry still hasn't settled on OpenAI-style formats for mid-run tool updates.
I wish they'd published the actual impact on model quality. Back-up-and-retry also gives 100% tool-call accuracy without polluting the context, and at 23/25 that's only +9% tokens on the tool call attempt, so if retry scores better on accuracy it might win out.
Those are two different questions: quality of the whole stack, versus how closely our setup matches the first-party provider. This post is about the second. Moonshot recommends constrained decoding on by default and serves Kimi that way, and OpenAI's newer Responses API has it on by default too. There is work showing it can hurt quality, depending on the model.
My prompts need around 300 tool calls each and I've been stuck with OpenAI and Anthropic since 2024. If anyone wonders what moat those two have: they nailed 100% tool calling via constrained decoding and prompt caching back then.
llama.cpp has done this forever. I was surprised that most other engines, vLLM included, didn't have it.
Does this work with Qwen 3.8?
Constrained decoding is available for Qwen 3.8 in vLLM, but mileage varies. Moonshot explicitly recommends turning it on for Kimi; I don't think that's the case for Qwen.