# NVIDIA NVRx on AWS EKS cuts GPU training downtime via async checkpointing

Digest AI · Hardware & Compute · published 2026-09-16T18:59:25Z

Canonical: https://digestai.news/story/nvidia-nvrx-on-aws-eks-cuts-gpu-training-downtime-via-async-checkpointing

## Summary

AWS has published a technical guide demonstrating how to integrate NVIDIA Resiliency Extension (NVRx) into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (EKS). The solution addresses two critical inefficiencies in large-scale distributed training: synchronous checkpointing that blocks all workers, and slow recovery from hardware or software faults that trigger cascading cluster restarts.

By adopting NVRx, teams can implement asynchronous checkpointing, which overlaps I/O operations with training steps. Benchmarks on H100 GPUs show this boosts training efficiency from roughly 60% to over 99%, eliminating the trade-off between frequent checkpoints and GPU utilization. Additionally, NVRx provides layered fault tolerance: in-process restarts handle soft faults like NCCL hangs in seconds, while the ftlauncher manages hard crashes like OOM kills. Compared to standard Kubernetes restarts, which can take over four minutes per fault, NVRx reduces recovery time to approximately 10 seconds, significantly improving infrastructure goodput for multi-node GPU workloads.

## Key points

- Async checkpointing via NVRx raises training efficiency from 60% to 99% by overlapping I/O with compute.
- In-process restart recovers from soft faults in ~10 seconds, drastically faster than Kubernetes' 270-second baseline.
- The solution uses NVIDIA H100s on AWS EKS with FSx for Lustre to support high-bandwidth distributed training.

## Why it matters

This integration significantly reduces the cost and time wasted on GPU idle time during large-scale AI training. By minimizing recovery windows and eliminating checkpoint bottlenecks, it enables more efficient use of expensive H100 clusters, making large model training more economically viable for enterprises.

## Sources

1. [Fault tolerant distributed training on Amazon EKS using NVRx](https://aws.amazon.com/blogs/machine-learning/fault-tolerant-distributed-training-on-amazon-eks-using-nvrx) (AWS Machine Learning Blog, 2026-09-16, primary source)

Part of the developing story: [The Race for Cheaper AI Inference Hardware](https://digestai.news/thread/3-8b-llm-trained-to-0-384-core-score-for-under-1-000-using-consumer-gpus) (8 stories)

## Cite

Digest AI, "NVIDIA NVRx on AWS EKS cuts GPU training downtime via async checkpointing", 16 September 2026, https://digestai.news/story/nvidia-nvrx-on-aws-eks-cuts-gpu-training-downtime-via-async-checkpointing

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/nvidia-nvrx-on-aws-eks-cuts-gpu-training-downtime-via-async-checkpointing.json