Supermicro, IBM and NVIDIA launch shared KV cache for long-context AI inference

The companies cite up to 56x faster time to first token on a 130,000-token prompt, based on prior benchmarking (June–August 2026). A 200-request test with 24M tokens and 825 GB of KV cache data showed throughput rising from 0.19 to 4.26 requests per second and total processing time dropping from 1,048.56 to 46.94 seconds. Under noisy-neighbor conditions, throughput dropped 18% but remained efficient. The architecture supports shared access across GPU nodes, non-disruptive scaling, and cache-aware routing via Dynamo’s KV Block Manager and NIXL transfer library. Deployment sizing depends on workloads, with Supermicro offering factory rack integration to cut deployment time from weeks to days.