# Amazon announces SageMaker HyperPod Inference Gateway to cut first-token latency up to 82%

Digest AI · Agents & Tools · published 2026-09-18T13:08:34Z · updated 2026-09-18T20:52:14Z

Canonical: https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok

## Summary

Amazon Web Services introduced the SageMaker HyperPod Inference Gateway, a Kubernetes‑native add‑on for EKS that routes large‑language‑model inference requests using real‑time GPU signals. The addon installs with a single command, requires no code changes, and works with any OpenAI‑compatible model server such as vLLM or TGI. In internal benchmarks the gateway lowered first‑token latency by up to 82%, turning a 4.4‑second wait into under 800 ms for a chatbot scenario.

Performance tests covered four models ranging from 8 billion to 235 billion parameters on H100‑based p5.48xlarge and A10G‑based g5 instances. Across three realistic traffic patterns—mixed GPU generations, bursty demand, and shared prompt prefixes—the gateway consistently outperformed default round‑robin load balancers, especially when hardware capacity varied or request spikes occurred. The system reports detailed metrics to Prometheus and CloudWatch, enabling observability and further tuning. Tier 1 per‑cluster routing is available today, with a global cross‑cluster router planned for a later release.

## Key points

- Up to 82% reduction in first‑token latency versus round‑robin routing.
- Deploys as a single EKS managed add‑on with no application code changes.
- Benchmarked on 8B‑235B models on H100 (p5.48xlarge) and A10G (g5) instances.

## Why it matters

Intelligent routing cuts GPU waste and latency, lowering inference costs and improving user experience for LLM‑powered applications.

## Sources

1. [Introducing Amazon SageMaker HyperPod Inference Gateway](https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-inference-gateway) (AWS Machine Learning Blog, 2026-09-18, primary source)
2. [Amazon SageMaker Inference: 2026 year-to-date launches in review](https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-inference-2026-year-to-date-launches-in-review) (AWS Machine Learning Blog, 2026-09-18, primary source)
3. [AWS Launches SageMaker HyperPod Inference Gateway for GPU-Aware Routing](https://unite.ai/aws-launches-sagemaker-hyperpod-inference-gateway-for-gpu-aware-routing) (Unite.AI, 2026-09-18)

Part of the developing story: [Amazon SageMaker Expands AI Inference Capabilities](https://digestai.news/thread/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-token) (2 stories)

## Cite

Digest AI, "Amazon announces SageMaker HyperPod Inference Gateway to cut first-token latency up to 82%", 18 September 2026, https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok.json