# Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

Digest AI · Agents & Tools · published 2026-10-19T16:03:23Z · updated 2026-10-19T16:05:17Z

Canonical: https://digestai.news/story/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-t

## Summary

The AWS Machine Learning Blog explains a bi-directional key‑value (KV) cache transfer that lets a decode node return previously computed KV blocks to a prefill node for the next turn of a multi‑turn agentic conversation. By reusing KV instead of recomputing it, the feature flattens time‑to‑first‑token (TTFT) growth as prompts become longer.

For Qwen3‑32B, latency rises from 158 ms at 2k tokens to 778 ms at 24k without KV reuse, but only from 122 ms to 460 ms with it. GLM‑5‑FP8 goes from 500 ms to 2,334 ms without reuse, versus 358 ms to 702 ms with reuse. The transfer adds a small 30‑130 ms overhead, far outweighed by the saved recompute time.

The implementation extends the NVIDIA Inference Xfer Library (NIXL) connector, adds a KV‑cache‑aware router, and uses a default block‑TTL of 480 seconds and a recompute threshold of 64 tokens. The feature is available in vLLM v0.21.0 or later and can be reproduced with the provided GitHub guide.

## Key points

- With KV reuse, Qwen3‑32B latency stays near 122 ms‑460 ms versus 158 ms‑778 ms without it; GLM‑5‑FP8 stays 358 ms‑702 ms versus 500 ms‑2,334 ms.
- Feature uses NIXL connector, a KV‑cache‑aware router, default block TTL 480 seconds and recompute threshold 64 tokens.

## Why it matters

Reducing first‑token latency for long multi‑turn conversations saves accelerator capacity and improves responsiveness of agentic AI applications at scale.

## Sources

1. [Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM](https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm) (AWS Machine Learning Blog, 2026-10-19, primary source)
2. [Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM](https://aws.amazon.com/blogs/machine-learning/push-based-kv-transfer-to-improve-ttft-and-tpot-for-disaggregated-inference-in-vllm) (AWS Machine Learning Blog, 2026-10-19, primary source)

Part of the developing story: [Accelerating Large Language Models Performance Chronicle](https://digestai.news/thread/rbs-attention-provides-radius-bounded-sparse-prefill-for-long-context-llms) (4 stories)

## Cite

Digest AI, "Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM", 19 October 2026, https://digestai.news/story/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-t

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-t.json