# Supermicro, IBM and NVIDIA launch shared KV cache for long-context AI inference

Digest AI · Hardware & Compute · published 2026-11-05T16:00:00Z

Canonical: https://digestai.news/story/more-performance-from-every-byte-how-full-stack-codesign-solves-ais-me

## Summary

Supermicro, IBM and NVIDIA announced a **KV cache storage solution** on November 5, 2026, to handle reusable AI inference context beyond GPU memory. The system combines Supermicro’s Petascale all-flash storage servers (using AMD EPYC 9005 processors), IBM Storage Scale software, NVIDIA’s Dynamo inference framework, and Spectrum-X Ethernet networking. It targets multi-turn applications, coding assistants, multimodal AI, and long-context retrieval-augmented generation, where large datasets (codebases, documents, images, or conversation histories) exceed GPU/CPU memory limits.

The companies cite **up to 56x faster time to first token** on a 130,000-token prompt, based on prior benchmarking (June–August 2026). A 200-request test with 24M tokens and 825 GB of KV cache data showed throughput rising from **0.19 to 4.26 requests per second** and total processing time dropping from **1,048.56 to 46.94 seconds**. Under noisy-neighbor conditions, throughput dropped **18%** but remained efficient. The architecture supports shared access across GPU nodes, non-disruptive scaling, and cache-aware routing via Dynamo’s KV Block Manager and NIXL transfer library. Deployment sizing depends on workloads, with Supermicro offering factory rack integration to cut deployment time from weeks to days.

## Key points

- Supermicro, IBM and NVIDIA launched a shared KV cache storage solution for long-context AI inference on November 5, 2026
- Benchmarks show up to 56x faster time to first token on 130,000-token prompts, with throughput improving from 0.19 to 4.26 requests per second
- System uses Supermicro Petascale servers, IBM Storage Scale, NVIDIA Dynamo, and Spectrum-X networking for multi-turn and agentic applications

## Why it matters

This storage-backed KV cache could reduce inference costs and latency for large-scale AI workloads by reusing context across requests, easing memory constraints for long-context models.

## Sources

1. [Supermicro Launches Shared KV Cache Storage for Long-Context AI](https://unite.ai/supermicro-launches-shared-kv-cache-storage-for-long-context-ai) (Unite.AI, 2026-11-05)
2. [More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge](https://blogs.nvidia.com/blog/memory-full-stack-codesign) (NVIDIA Blog, 2026-11-05, primary source)

## Cite

Digest AI, "Supermicro, IBM and NVIDIA launch shared KV cache for long-context AI inference", 5 November 2026, https://digestai.news/story/more-performance-from-every-byte-how-full-stack-codesign-solves-ais-me

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/more-performance-from-every-byte-how-full-stack-codesign-solves-ais-me.json