Amazon SageMaker HyperPod Introduces Model Caching to Cut Cold Starts
Amazon Web Services has added model caching to its SageMaker HyperPod inference platform, allowing large language models to start serving traffic in seconds instead of minutes. The new feature pre‑loads both the inference server container image and the model weights onto each node’s NVMe storage, eliminating the 5–7 minute ECR pull and the 20‑30 minute download from S3 or other backends that…
Key points
- Model caching pre‑loads weights to NVMe, cutting 600+ GB cold starts from 30 min to seconds.
- Weights cache speeds scale‑out 60 % faster; image cache reduces pull time by up to 97 %.
- Caching works across S3, FSx, and HuggingFace, managed by two CRDs in SageMaker HyperPod.
Benchmarks show that enabling the weights cache speeds up scale‑out by roughly 60 % for models between 57 GB and 145 GB, while the image cache can cut the image‑pull latency by up to 97 %. For a 600 GB model like DeepSeek‑R1, the caching mechanism removes a 30‑minute download, making new pods ready in a matter of seconds.
The feature is delivered through two custom resource definitions—ModelDataCacheConfig and ModelImageCache—managed automatically by the HyperPod Inference Operator, and it works across all supported storage backends, including Amazon S3, FSx for Lustre, and HuggingFace Hub.
The story so far
10 episodes →- Amazon SageMaker HyperPod Introduces Model Caching to Cut Cold Startsthis story
Reduce inference cold starts on Amazon SageMaker HyperPod with model caching
AWS Machine Learning Blog · 10 September 2026
Loading the full article…
This text was published by AWS Machine Learning Blog and written by Kareem Syed-Mohammed. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Enterprise & Industry
All →- Cyara’s CEO outlines AI agent oversight as a workforce model · 1 src
- Arcana adds institutional portfolio data to Claude’s connectors · 1 src
- Anthropic brings Claude inference to India via Amazon Bedrock · 6 src
- Infor adds industry-specific AI agents to Velocity Suite, citing generic AI gaps · 1 src
- Tech Mahindra trains 12,500+ staff on Google’s Gemini Enterprise agents · 1 src
Comments
via GitHub Discussions