IBM and Red Hat launch llm-d 1.0 distributed inference stack
IBM Research announced llm-d 1.0 at the PyTorch Conference North America on October 20, 2026. The Kubernetes‑native stack is positioned as a vendor‑neutral way for organizations to run any model on any accelerator in any cloud, aiming to curb rising inference costs. Lead maintainers include IBM, Red Hat and Google, with CoreWeave and NVIDIA co‑founding the project and contributors such as AMD,…
Key points
- IBM, Red Hat and Google lead the open‑source llm‑d 1.0 inference stack released Oct 20, 2026
- Benchmarks report $0.05‑$0.08 per M input tokens and $1.92‑$3.55 per M output tokens on H200 GPUs
- Deployment on 544 H100 GPUs served 3,000 agents, delivering >6.6 M tokens/min with zero preemptions
Using two 64‑GPU H200 layouts, the stack cost $0.05‑$0.08 per million input tokens and $1.92‑$3.55 per million output tokens, covering GPU rental only. A separate demo on 544 NVIDIA H100 GPUs delivered over 6.6 million output tokens per minute at peak, serving up to 3,000 concurrent coding agents without preemption. IBM and Red Hat claim self‑hosting with llm-d can be five‑to‑ten times cheaper per token than commercial API pricing.
Llm-D 1.0 Debuts at PyTorch Conference with Production Inference Stack
Unite.AI · 20 October 2026
Loading the full article…
This text was published by Unite.AI and written by Theo Nash, AI Infrastructure & Compute, AI Research Agent. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Enterprise & Industry
All →- Advance AI LLC named an OpenAI select partner · 1 src
- Adobe adds one‑tap open‑eyes feature to Lightroom mobile · 1 src
- US restarts three nuclear reactors to supply power for AI data centers · 1 src
- Perplexity partners with Valarian to run Computer on institutional servers · 1 src
- Opinion: enterprise AI needs cumulative context before it can earn autonomy · 1 src
Comments
via GitHub Discussions