NVIDIA NVRx on AWS EKS cuts GPU training downtime via async checkpointing
AWS has published a technical guide demonstrating how to integrate NVIDIA Resiliency Extension (NVRx) into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (EKS). The solution addresses two critical inefficiencies in large-scale distributed training: synchronous checkpointing that blocks all workers, and slow recovery from hardware or software faults that…
Key points
- Async checkpointing via NVRx raises training efficiency from 60% to 99% by overlapping I/O with compute.
- In-process restart recovers from soft faults in ~10 seconds, drastically faster than Kubernetes' 270-second baseline.
- The solution uses NVIDIA H100s on AWS EKS with FSx for Lustre to support high-bandwidth distributed training.
By adopting NVRx, teams can implement asynchronous checkpointing, which overlaps I/O operations with training steps. Benchmarks on H100 GPUs show this boosts training efficiency from roughly 60% to over 99%, eliminating the trade-off between frequent checkpoints and GPU utilization. Additionally, NVRx provides layered fault tolerance: in-process restarts handle soft faults like NCCL hangs in seconds, while the ftlauncher manages hard crashes like OOM kills. Compared to standard Kubernetes restarts, which can take over four minutes per fault, NVRx reduces recovery time to approximately 10 seconds, significantly improving infrastructure goodput for multi-node GPU workloads.
The story so far
8 episodes →- NVIDIA NVRx on AWS EKS cuts GPU training downtime via async checkpointingthis story
Fault tolerant distributed training on Amazon EKS using NVRx
AWS Machine Learning Blog · 16 September 2026
Loading the full article…
This text was published by AWS Machine Learning Blog and written by Aravind Neelakantan. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Hardware & Compute
All →- Nvidia, AMD, Micron drop premarket as AI financing fears rise · 4 src
- Apple delays Apple Intelligence launch in China to December 2026 · 2 src
- Liteon unveils 800 VDC power tech for NVIDIA’s AI infrastructure at OCP summit · 1 src
- Mindbeam announces collaboration with NVIDIA on Litespark Nemotron 3.5 Super VL model and inference engine · 4 src
- Arista Networks launches 7060XE7 Series for AI data center networking · 1 src
Comments
via GitHub Discussions