Agents & Tools19 October 2026, 16:03 UTCupdated 19 October 2026, 16:05 UTC8 min read

Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

The AWS Machine Learning Blog explains a bi-directional key‑value (KV) cache transfer that lets a decode node return previously computed KV blocks to a prefill node for the next turn of a multi‑turn agentic conversation. By reusing KV instead of recomputing it, the feature flattens time‑to‑first‑token (TTFT) growth as prompts become longer.

2 sources primary source

Key points

With KV reuse, Qwen3‑32B latency stays near 122 ms‑460 ms versus 158 ms‑778 ms without it; GLM‑5‑FP8 stays 358 ms‑702 ms versus 500 ms‑2,334 ms.
Feature uses NIXL connector, a KV‑cache‑aware router, default block TTL 480 seconds and recompute threshold 64 tokens.

For Qwen3‑32B, latency rises from 158 ms at 2k tokens to 778 ms at 24k without KV reuse, but only from 122 ms to 460 ms with it. GLM‑5‑FP8 goes from 500 ms to 2,334 ms without reuse, versus 358 ms to 702 ms with reuse. The transfer adds a small 30‑130 ms overhead, far outweighed by the saved recompute time.

The implementation extends the NVIDIA Inference Xfer Library (NIXL) connector, adds a KV‑cache‑aware router, and uses a default block‑TTL of 480 seconds and a recompute threshold of 64 tokens. The feature is available in vLLM v0.21.0 or later and can be reproduced with the provided GitHub guide.

The story so far

4 episodes →

19 Oct
Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLMthis story
14 Oct
Researchers introduce Echo for faster 4-bit LLM training with low error
1 Oct
Researchers propose DLFP controller to cut AI inference latency by up to 30%
21 Sept
RBS-Attention provides Radius-Bounded sparse prefill for Long-Context LLMs

Full story from AWS Machine Learning Blog · by Sunita Nadampalli primary sourceOpen source ↗