# Amazon SageMaker HyperPod Introduces Model Caching to Cut Cold Starts

Digest AI · Enterprise & Industry · published 2026-09-10T21:37:49Z

Canonical: https://digestai.news/story/amazon-sagemaker-hyperpod-introduces-model-caching-to-cut-cold-starts

## Summary

Amazon Web Services has added model caching to its SageMaker HyperPod inference platform, allowing large language models to start serving traffic in seconds instead of minutes. The new feature pre‑loads both the inference server container image and the model weights onto each node’s NVMe storage, eliminating the 5–7 minute ECR pull and the 20‑30 minute download from S3 or other backends that previously dominated cold‑start times.

Benchmarks show that enabling the weights cache speeds up scale‑out by roughly 60 % for models between 57 GB and 145 GB, while the image cache can cut the image‑pull latency by up to 97 %. For a 600 GB model like DeepSeek‑R1, the caching mechanism removes a 30‑minute download, making new pods ready in a matter of seconds.

The feature is delivered through two custom resource definitions—ModelDataCacheConfig and ModelImageCache—managed automatically by the HyperPod Inference Operator, and it works across all supported storage backends, including Amazon S3, FSx for Lustre, and HuggingFace Hub.

## Key points

- Model caching pre‑loads weights to NVMe, cutting 600+ GB cold starts from 30 min to seconds.
- Weights cache speeds scale‑out 60 % faster; image cache reduces pull time by up to 97 %.
- Caching works across S3, FSx, and HuggingFace, managed by two CRDs in SageMaker HyperPod.

## Why it matters

By slashing inference cold‑start latency, enterprises can deploy larger models with tighter scaling, improving user experience and reducing operational costs in AI‑driven services.

## Sources

1. [Reduce inference cold starts on Amazon SageMaker HyperPod with model caching](https://aws.amazon.com/blogs/machine-learning/reduce-inference-cold-starts-on-amazon-sagemaker-hyperpod-with-model-caching) (AWS Machine Learning Blog, 2026-09-10, primary source)

Part of the developing story: [AWS Optimizes Enterprise AI Infrastructure](https://digestai.news/thread/amazon-sagemaker-hyperpod-introduces-model-caching-to-cut-cold-starts) (10 stories)

## Cite

Digest AI, "Amazon SageMaker HyperPod Introduces Model Caching to Cut Cold Starts", 10 September 2026, https://digestai.news/story/amazon-sagemaker-hyperpod-introduces-model-caching-to-cut-cold-starts

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/amazon-sagemaker-hyperpod-introduces-model-caching-to-cut-cold-starts.json