AI lab replaces scheduler to cut GPU squatting and on-call toil
The AI Infrastructure team at a research lab manages thousands of NVIDIA GPUs—H100, B200, and B300—across clusters of 88 to 1024 units. Demand for GPU time exceeds supply by 2 to 3 times, leading to inefficiencies like ‘squatting’ (parking idle workloads) and priority inflation, where all jobs defaulted to ‘HIGH’ priority. The team replaced a priority-based scheduler with a system using GPU time…
Key points
- Replaced priority-based scheduler with GPU time budgets and hierarchical fair-share allocation to curb squatting and priority inflation
- Teams received 98% of their owed GPU hours over 30 days, with on-call repairs dropping by 74%
- Debug workloads now wait 30 seconds vs. 6 hours previously, but some interactive sessions face preemption after 8 hours
The new system ties GPU allocation to research program strategy, with managers setting budgets for projects. Workloads must declare a minimum runtime (up to 8 hours) to avoid preemption. Over a 30-day test, teams received 98% of their owed GPU hours, and on-call repairs requiring human intervention dropped by 74%. Debug workloads saw queue wait times drop from 6 hours to 30 seconds. However, some interactive sessions now face preemption after 8 hours, prompting plans for a CPU-only cluster and restorable sessions.
Impactful scheduling for GPU clusters
Hugging Face · 9 October 2026
Loading the full article…
This text was published by Hugging Face. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Hardware & Compute
All →- Microsoft expands Azure Maia system to multi-gigawatt deployments starting in 2027 · 3 src
- Apple delays HomeView EU launch due to Siri’s unavailability · 1 src
- Apple unveils HomeView demo with Siri and facial recognition for up to six users · 1 src
- Apple adds Siri AI and Wi-Fi 7 to fourth-gen Apple TV 4K for $299 · 1 src
- Satlyt raises $8M to run AI on satellites with Google DeepMind’s Gemma · 1 src
Comments
via GitHub Discussions