Amazon SageMaker Inference supports GPU partitioning

The feature works on endpoints that use inference components. A partition receives a fixed share of compute units and memory, with hardware isolation. Customers can choose a fraction of a newer accelerator instead of a whole one. The article notes that provisioning with MIG adds roughly 17 seconds to endpoint startup, but otherwise the rest of the launch time is unchanged.

Salesforce helped define requirements and validated the initial implementation. Richard Chang, Software Engineering Architect at Salesforce, said the capability would allow them to consolidate workloads safely, improve GPU utilization, reduce inference costs, and maintain predictable performance. The maximum number of partitions depends on the accelerator and never exceeds seven. MIG partitions a GPU into up to seven instances, each with its own compute, cache, memory and bandwidth.