{"version":1,"type":"story","url":"https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok","json":"https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok.json","markdown":"https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok.md","slug":"amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok","headline":"Amazon announces SageMaker HyperPod Inference Gateway to cut first-token latency up to 82%","summary":"Amazon Web Services introduced the SageMaker HyperPod Inference Gateway, a Kubernetes‑native add‑on for EKS that routes large‑language‑model inference requests using real‑time GPU signals. The addon installs with a single command, requires no code changes, and works with any OpenAI‑compatible model server such as vLLM or TGI. In internal benchmarks the gateway lowered first‑token latency by up to 82%, turning a 4.4‑second wait into under 800 ms for a chatbot scenario.\n\nPerformance tests covered four models ranging from 8 billion to 235 billion parameters on H100‑based p5.48xlarge and A10G‑based g5 instances. Across three realistic traffic patterns—mixed GPU generations, bursty demand, and shared prompt prefixes—the gateway consistently outperformed default round‑robin load balancers, especially when hardware capacity varied or request spikes occurred. The system reports detailed metrics to Prometheus and CloudWatch, enabling observability and further tuning. Tier 1 per‑cluster routing is available today, with a global cross‑cluster router planned for a later release.","keyPoints":["Up to 82% reduction in first‑token latency versus round‑robin routing.","Deploys as a single EKS managed add‑on with no application code changes.","Benchmarked on 8B‑235B models on H100 (p5.48xlarge) and A10G (g5) instances."],"whyItMatters":"Intelligent routing cuts GPU waste and latency, lowering inference costs and improving user experience for LLM‑powered applications.","category":{"slug":"agents","name":"Agents & Tools","url":"https://digestai.news/category/agents"},"entities":{"companies":["Amazon","AWS"],"models":[],"people":[]},"firstPublishedAt":"2026-09-18T13:08:34Z","updatedAt":"2026-09-18T20:52:14Z","sourceCount":3,"hasPrimarySource":true,"sources":[{"outlet":"AWS Machine Learning Blog","title":"Introducing Amazon SageMaker HyperPod Inference Gateway","url":"https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-inference-gateway","publishedAt":"2026-09-18T13:08:34Z","type":"primary","primary":true,"lead":true},{"outlet":"AWS Machine Learning Blog","title":"Amazon SageMaker Inference: 2026 year-to-date launches in review","url":"https://aws.amazon.com/blogs/machine-learning/amazon-sagemaker-inference-2026-year-to-date-launches-in-review","publishedAt":"2026-09-18T20:52:14Z","type":"primary","primary":true,"lead":false},{"outlet":"Unite.AI","title":"AWS Launches SageMaker HyperPod Inference Gateway for GPU-Aware Routing","url":"https://unite.ai/aws-launches-sagemaker-hyperpod-inference-gateway-for-gpu-aware-routing","publishedAt":"2026-09-18T13:27:01Z","type":"press","primary":false,"lead":false}],"sourceNotes":null,"discussions":[],"thread":{"title":"Amazon SageMaker Expands AI Inference Capabilities","url":"https://digestai.news/thread/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-token","storyCount":2},"cite":{"text":"Digest AI, \"Amazon announces SageMaker HyperPod Inference Gateway to cut first-token latency up to 82%\", 18 September 2026, https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok","publisher":"Digest AI","title":"Amazon announces SageMaker HyperPod Inference Gateway to cut first-token latency up to 82%","datePublished":"2026-09-18T13:08:34Z","url":"https://digestai.news/story/amazon-announces-sagemaker-hyperpod-inference-gateway-to-cut-first-tok"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}