AWS introduces concurrency sweeps in SageMaker AI to right‑size generative endpoints
Amazon Web Services released a new capacity‑planning feature called concurrency sweeps, built into SageMaker AI Inference Recommendations. The workflow benchmarks a generative model—NVIDIA Nemotron‑3 Nano 30B—deployed on an ml.g7e.2xlarge instance that uses an NVIDIA Blackwell GPU. By sending controlled levels of simultaneous traffic (e.g., 64, 256, 1,024 requests) the sweep measures throughput…
Key points
- Concurrency sweeps in SageMaker AI benchmark generative models like NVIDIA Nemotron‑3 Nano 30B on ml.g7e.2xlarge instances.
- In the example, throughput plateaus at 256 concurrent requests; SLA of 50 s p99 latency met at 320 concurrency with 2,782 tokens/sec.
- Feature uses the CreateAIBenchmarkJob API and can run automated max‑concurrency‑under‑SLA searches.
In the example notebook, throughput flattens at 256 concurrent requests, while the p99 latency crosses the service‑level agreement (SLA) threshold at higher loads. Using the optional “max‑concurrency‑under‑SLA” search recipe, the tool automatically finds the highest concurrency that keeps p99 latency under 50 seconds, landing at 320 concurrent requests with 2,782 tokens per second. Results are saved to Amazon S3, and the endpoint can be right‑sized based on the identified optimal concurrency and instance count. The feature requires only an AWS account, appropriate IAM role, and a service quota for the chosen instance type.
Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI
AWS Machine Learning Blog · 22 September 2026
Loading the full article…
This text was published by AWS Machine Learning Blog and written by Mona Mona. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Enterprise & Industry
All →- Google makes Gemini study notebooks available for Workspace accounts · 1 src
- Salesforce launches enterprise AI harness to streamline small business operations · 1 src
- Microsoft launches fourth India cloud region in Hyderabad with zero-water cooling · 10 src
- Palo Alto Networks launches Unit 42 AI Defense using Anthropic Mythos and OpenAI GPT-5.6 · 4 src
- UTMB embeds OpenEvidence AI platform in its electronic health record · 1 src
Comments
via GitHub Discussions