vLLM 0.31 adds constrained decoding for 100% tool-call accuracy
Ubicloud found that open-weight models like Kimi K2.6 and DeepSeek V4 Flash often fail to generate correct tool calls, with vLLM version 0.28 achieving only 92% schema accuracy for Kimi K2.6. This led to nonsensical context and lower-quality results, especially in agentic workloads where multi-turn sessions are common. The issue stems from probabilistic token generation, where models may produce…
Key points
- Constrained decoding in vLLM 0.31 guarantees 100% schema accuracy for Kimi and DeepSeek
- Ubicloud enabled the feature via `--tool-strict-level` flag after vLLM disabled it by default
However, vLLM disabled constrained decoding by default due to model-agnostic concerns, dropping Ubicloud’s accuracy to 91%. The company enabled the feature explicitly in vLLM 0.31 via the --tool-strict-level flag, achieving 100% schema accuracy for Kimi and DeepSeek. The technique uses XGrammar to restrict token sampling to valid paths, ensuring correct tool calls without significant performance overhead.
The story so far
2 episodes →- vLLM 0.31 adds constrained decoding for 100% tool-call accuracythis story
Fixing Tool-Call Accuracy in vLLM
ubicloud.com · 20 October 2026
Loading the full article…
This text was published by ubicloud.com. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Hacker News discussion · 47 pointsnews.ycombinator.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Agents & Tools
All →- Anthropic launches Claude Duet agent; Mythos 5 sent malicious code to developer during test · 5 src
- OpenAI Agents SDK lets Python developers build web-searching, tool-using agents · 1 src
- AI agents navigate software via clicks and typing without APIs · 1 src
- Claude Code AI detects cryptominer on Mac during temperature monitor build · 1 src
- Google releases ML Drift GPU inference engine as open-source · 1 src
Comments
via GitHub Discussions