{"version":1,"type":"story","url":"https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil","json":"https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil.json","markdown":"https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil.md","slug":"ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil","headline":"AI lab replaces scheduler to cut GPU squatting and on-call toil","summary":"The AI Infrastructure team at a research lab manages thousands of NVIDIA GPUs—H100, B200, and B300—across clusters of 88 to 1024 units. Demand for GPU time exceeds supply by 2 to 3 times, leading to inefficiencies like ‘squatting’ (parking idle workloads) and priority inflation, where all jobs defaulted to ‘HIGH’ priority. The team replaced a priority-based scheduler with a system using GPU time budgets, hierarchical fair-share allocation, and time-slicing contracts.\n\nThe new system ties GPU allocation to research program strategy, with managers setting budgets for projects. Workloads must declare a minimum runtime (up to 8 hours) to avoid preemption. Over a 30-day test, teams received 98% of their owed GPU hours, and on-call repairs requiring human intervention dropped by 74%. Debug workloads saw queue wait times drop from 6 hours to 30 seconds. However, some interactive sessions now face preemption after 8 hours, prompting plans for a CPU-only cluster and restorable sessions.","keyPoints":["Replaced priority-based scheduler with GPU time budgets and hierarchical fair-share allocation to curb squatting and priority inflation","Teams received 98% of their owed GPU hours over 30 days, with on-call repairs dropping by 74%","Debug workloads now wait 30 seconds vs. 6 hours previously, but some interactive sessions face preemption after 8 hours"],"whyItMatters":"This approach shifts GPU allocation from ad-hoc prioritization to strategic budgeting, reducing waste and human intervention while improving fairness and responsiveness for smaller tasks.","category":{"slug":"hardware","name":"Hardware & Compute","url":"https://digestai.news/category/hardware"},"entities":{"companies":["Hugging Face"],"models":[],"people":[]},"firstPublishedAt":"2026-10-09T15:20:29Z","updatedAt":"2026-10-09T15:20:29Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"Hugging Face","title":"Impactful scheduling for GPU clusters","url":"https://huggingface.co/blog/allenai/impactful-scheduling","publishedAt":"2026-10-09T15:20:29Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"AI lab replaces scheduler to cut GPU squatting and on-call toil\", 9 October 2026, https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil","publisher":"Digest AI","title":"AI lab replaces scheduler to cut GPU squatting and on-call toil","datePublished":"2026-10-09T15:20:29Z","url":"https://digestai.news/story/ai-lab-replaces-scheduler-to-cut-gpu-squatting-and-on-call-toil"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}