BlockLock fixes LLM inference batching for reproducible outputs
BlockLock is a new method for ensuring that large language model (LLM) inference produces the same output for a given request, no matter what other requests are in its batch. Existing approaches either constrain every operator in the batch or isolate prefill stages, which limits shared compute or requires padding decode stages. BlockLock instead fixes the geometry of the protected request while…
Key points
- BlockLock fixes LLM inference geometry to guarantee reproducible outputs per request, even in batched deployments
- Two modes tested: BlockLock-Rows (fixed-width rows) and BlockLock-Pack (terminal-anchored packing with dummy segment)
- Latency drops **53.5–55.0%** and throughput rises **115.2–122.2%** vs. vLLM Batch-Invariant on five models
The authors propose two modes: BlockLock-Rows, which uses fixed-width rows, and BlockLock-Pack, which isolates a dummy segment and extracts the target request’s KV state for decode. Tests on five dense models—Qwen3-8B/14B, Llama-3.1-8B-Instruct, and Gemma-3-4B/12B-IT—across three GPUs (NVIDIA A6000, A100, H100) show both modes preserve target sequences and Top20-logit bits. BlockLock-Rows cuts latency by 53.5–55.0% and boosts throughput by 115.2–122.2% over vLLM’s batch-invariant method, without requiring custom kernel implementations.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers link AI agent power to human Goal attainment in formal models · 1 src
- PromptOps framework proposed for iterative LLM judge optimization · 1 src
- Researchers introduce BCSS to scale symbolic models of reasoning · 1 src
- Researchers test Cross-Agent formalization for clinical trial criteria · 1 src
- Researchers introduce efxplorer for Cooperative Multi-Agent reinforcement learning · 1 src
Comments
via GitHub Discussions