Buoyant releases kinference, an open source inference control plane
buoyant released kinference, an open‑source inference control plane for kubernetes that sits in front of vllm workers. the project, written in rust and licensed under apache 2.0, is available on github as v0.1.0‑alpha.7 and is configured through five kubernetes custom resource definitions. it claims to route requests, enforce token‑rate limits, perform kv‑cache‑aware load balancing, and expose…
Key points
- buoyant releases kinference, an open‑source inference control plane for kubernetes and vllm
- self‑reported median ttft drops 95 % on shared‑prefix traffic, from 6.88 s to 0.31 s
benchmark results, self‑reported by buoyant, show a median ttft drop from 6.88 s to 0.31 s (95 %) on shared‑prefix traffic, while no improvement appears on prompts without shared prefixes. the system uses three l40s GPUs per pool and relies on nvidia’s dynamo‑kv‑router crate for cache events.
kinference is still in alpha; rate limiting, classifiers, and anthropic api support are not yet shipped. deployment requires installing gateway api v1.5.1 crds, the buoyant gateway helm chart, and the kinference chart, after which prometheus metrics and grafana dashboards are available.
Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
MarkTechPost · 7 November 2026
Loading the full article…
This text was published by MarkTechPost and written by Michal Sutter. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Enterprise & Industry
All →- NVIDIA releases open-source auto ontology to link databases to AI agents · 1 src
- NVIDIA proposes converting telecom facilities into AI hubs · 3 src
- Anthropic adds enterprise security checks to Claude with inference hooks · 1 src
- BMO deploys AI agent to speed credit card call prep by 20–30% · 1 src
- OpenEvidence integrates ACR patient fact sheets into clinical AI workflows · 1 src
Comments
via GitHub Discussions