GPU utilisation shows compute activity, not whether capacity is actually available

Kubernetes assigns whole GPUs to pods by default, so a single workload can block all others even if it uses only a fraction of the device. Memory, placement rules, and application bottlenecks also cause queues. Cast AI supports time‑slicing, MIG, and MPS sharing methods, each with trade‑offs in memory isolation, hardware compatibility, and observability. Diagnosing the root cause—allocation state, memory occupancy, placement constraints, or application bottlenecks—requires a specific sequence of checks.

INT4‑quantised 7‑B models use ~4 GB, and INT8 cuts requirements roughly in half. These figures illustrate why memory‑based sharing can be more effective than time‑slicing when VRAM is the limiting factor.