{"version":1,"type":"story","url":"https://digestai.news/story/buoyant-releases-kinference-an-open-source-inference-control-plane","json":"https://digestai.news/story/buoyant-releases-kinference-an-open-source-inference-control-plane.json","markdown":"https://digestai.news/story/buoyant-releases-kinference-an-open-source-inference-control-plane.md","slug":"buoyant-releases-kinference-an-open-source-inference-control-plane","headline":"Buoyant releases kinference, an open source inference control plane","summary":"buoyant released kinference, an open‑source inference control plane for kubernetes that sits in front of vllm workers. the project, written in rust and licensed under apache 2.0, is available on github as v0.1.0‑alpha.7 and is configured through five kubernetes custom resource definitions. it claims to route requests, enforce token‑rate limits, perform kv‑cache‑aware load balancing, and expose per‑tenant metrics.\n\nbenchmark results, self‑reported by buoyant, show a median ttft drop from 6.88 s to 0.31 s (95 %) on shared‑prefix traffic, while no improvement appears on prompts without shared prefixes. the system uses three l40s GPUs per pool and relies on nvidia’s dynamo‑kv‑router crate for cache events.\n\nkinference is still in alpha; rate limiting, classifiers, and anthropic api support are not yet shipped. deployment requires installing gateway api v1.5.1 crds, the buoyant gateway helm chart, and the kinference chart, after which prometheus metrics and grafana dashboards are available.","keyPoints":["buoyant releases kinference, an open‑source inference control plane for kubernetes and vllm","self‑reported median ttft drops 95 % on shared‑prefix traffic, from 6.88 s to 0.31 s"],"whyItMatters":"kinference gives dev teams a policy‑driven way to manage multi‑tenant inference on kubernetes, potentially reducing latency and increasing throughput for large‑model workloads. its open‑source nature and integration with vllm could lower operational costs for enterprises deploying LLMs at scale.","category":{"slug":"enterprise","name":"Enterprise & Industry","url":"https://digestai.news/category/enterprise"},"entities":{"companies":["Buoyant","NVIDIA","Bedrock","Anthropic","GuideLLM"],"models":["vLLM","Qwen3.8-27B-FP8","llm-d","Dynamo"],"people":["Michal Sutter"]},"firstPublishedAt":"2026-11-07T08:16:33Z","updatedAt":"2026-11-07T08:16:33Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"MarkTechPost","title":"Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM","url":"https://marktechpost.com/2026/11/07/buoyant-kinference-open-source-inference-control-plane-kubernetes-vllm","publishedAt":"2026-11-07T08:16:33Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Buoyant releases kinference, an open source inference control plane\", 7 November 2026, https://digestai.news/story/buoyant-releases-kinference-an-open-source-inference-control-plane","publisher":"Digest AI","title":"Buoyant releases kinference, an open source inference control plane","datePublished":"2026-11-07T08:16:33Z","url":"https://digestai.news/story/buoyant-releases-kinference-an-open-source-inference-control-plane"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}