# AI lab replaces scheduler to cut GPU squatting and on-call toil

Digest AI · Hardware & Compute · published 2026-10-09T15:20:29Z

Canonical: https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil

## Summary

The AI Infrastructure team at a research lab manages thousands of NVIDIA GPUs—H100, B200, and B300—across clusters of 88 to 1024 units. Demand for GPU time exceeds supply by 2 to 3 times, leading to inefficiencies like ‘squatting’ (parking idle workloads) and priority inflation, where all jobs defaulted to ‘HIGH’ priority. The team replaced a priority-based scheduler with a system using GPU time budgets, hierarchical fair-share allocation, and time-slicing contracts.

The new system ties GPU allocation to research program strategy, with managers setting budgets for projects. Workloads must declare a minimum runtime (up to 8 hours) to avoid preemption. Over a 30-day test, teams received 98% of their owed GPU hours, and on-call repairs requiring human intervention dropped by 74%. Debug workloads saw queue wait times drop from 6 hours to 30 seconds. However, some interactive sessions now face preemption after 8 hours, prompting plans for a CPU-only cluster and restorable sessions.

## Key points

- Replaced priority-based scheduler with GPU time budgets and hierarchical fair-share allocation to curb squatting and priority inflation
- Teams received 98% of their owed GPU hours over 30 days, with on-call repairs dropping by 74%
- Debug workloads now wait 30 seconds vs. 6 hours previously, but some interactive sessions face preemption after 8 hours

## Why it matters

This approach shifts GPU allocation from ad-hoc prioritization to strategic budgeting, reducing waste and human intervention while improving fairness and responsiveness for smaller tasks.

## Sources

1. [Impactful scheduling for GPU clusters](https://huggingface.co/blog/allenai/impactful-scheduling) (Hugging Face, 2026-10-09, primary source)

## Cite

Digest AI, "AI lab replaces scheduler to cut GPU squatting and on-call toil", 9 October 2026, https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil.json