Ggml-org adds `--spec-draft-n-ubatch` to llama.cpp for batch control
llama.cpp now lets users set a custom physical batch size for draft context via the --spec-draft-n-ubatch flag. The change, merged in a pull request by ggml-org, allows drafts to process tokens in smaller chunks than the target model’s batch size. This reduces VRAM usage for the draft’s compute buffer while preserving the target’s KV cache efficiency. The default remains unchanged, matching the…
Key points
- New `--spec-draft-n-ubatch` flag lets users set draft batch size independently of target model
- Smaller values shrink VRAM usage for draft compute buffer while preserving target KV cache
- Default behavior unchanged; flag is optional and requires no code changes
The update is part of ongoing optimizations for running large language models efficiently on limited hardware. It does not alter the draft’s final output but may improve performance for users working with constrained memory. The change was assisted by Qwen3.8-Flash-Next and documented in the repository’s test suite.
The story so far
4 episodes →- Ggml-org adds `--spec-draft-n-ubatch` to llama.cpp for batch controlthis story
common : add --spec-draft-n-ubatch by ggerganov · Pull Request #30459 · ggml-org/llama.cpp
github.com · 14 October 2026Loading the full article…
This text was published by github.com. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Reddit discussionreddit.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic warns small AI models may self-replicate on IoT devices by end of 2026 · 19 src
- Google DeepMind’s AI forecasted Hurricane Isaias’s path 19 miles off · 1 src
- Hospitals, banks and edge devices use federated Voting for feature selection · 1 src
- Researchers introduce Echo for faster 4-bit LLM training with low error · 2 src
- Danaher and NVIDIA team to embed AI in scientific instruments · 2 src
Comments
via GitHub Discussions