# AWS introduces concurrency sweeps in SageMaker AI to right‑size generative endpoints

Digest AI · Enterprise & Industry · published 2026-09-22T15:35:53Z

Canonical: https://digestai.news/story/aws-introduces-concurrency-sweeps-in-sagemaker-ai-to-rightsize-generat

## Summary

Amazon Web Services released a new capacity‑planning feature called concurrency sweeps, built into SageMaker AI Inference Recommendations. The workflow benchmarks a generative model—NVIDIA Nemotron‑3 Nano 30B—deployed on an ml.g7e.2xlarge instance that uses an NVIDIA Blackwell GPU. By sending controlled levels of simultaneous traffic (e.g., 64, 256, 1,024 requests) the sweep measures throughput (tokens per second) and latency, identifying the saturation point where adding more concurrency no longer improves throughput and latency spikes.

In the example notebook, throughput flattens at 256 concurrent requests, while the p99 latency crosses the service‑level agreement (SLA) threshold at higher loads. Using the optional “max‑concurrency‑under‑SLA” search recipe, the tool automatically finds the highest concurrency that keeps p99 latency under 50 seconds, landing at 320 concurrent requests with 2,782 tokens per second. Results are saved to Amazon S3, and the endpoint can be right‑sized based on the identified optimal concurrency and instance count. The feature requires only an AWS account, appropriate IAM role, and a service quota for the chosen instance type.

## Key points

- Concurrency sweeps in SageMaker AI benchmark generative models like NVIDIA Nemotron‑3 Nano 30B on ml.g7e.2xlarge instances.
- In the example, throughput plateaus at 256 concurrent requests; SLA of 50 s p99 latency met at 320 concurrency with 2,782 tokens/sec.
- Feature uses the CreateAIBenchmarkJob API and can run automated max‑concurrency‑under‑SLA searches.

## Why it matters

Gives AWS customers a data‑driven method to size generative AI endpoints, cutting idle GPU costs and avoiding latency spikes in production.

## Sources

1. [Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI](https://aws.amazon.com/blogs/machine-learning/right-size-generative-ai-endpoints-with-concurrency-sweeps-on-amazon-sagemaker-ai) (AWS Machine Learning Blog, 2026-09-22, primary source)

## Cite

Digest AI, "AWS introduces concurrency sweeps in SageMaker AI to right‑size generative endpoints", 22 September 2026, https://digestai.news/story/aws-introduces-concurrency-sweeps-in-sagemaker-ai-to-rightsize-generat

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/aws-introduces-concurrency-sweeps-in-sagemaker-ai-to-rightsize-generat.json