IBM makes Spyre a first-class PyTorch backend

Key Features of Spyre

Spyre is described as an AI accelerator optimized for inference and designed for enterprise teams. It targets matrix-heavy work in language generation and embedding models. Unlike a conventional GPU, which executes kernels as instruction streams across scheduled groups of threads, Spyre triggers computation as data arrives at its compute engines. The integration with PyTorch involved updates to the compiler stack, including moving torch-spyre from sendnn (a custom compilation stack) to backend=inductor (an open-source extension). This move reduced the size and complexity of the codebase.

Performance Improvements

The post detailed performance improvements achieved through Spyre integration. These include a 29–69% reduction in matmul kernel time, a drop from 32.51 microseconds to 23.67 microseconds for softmax kernels (a 27% reduction), and the introduction of a fused MLP-style kernel that improved core utilization and reduced kernel times.

Native Device Integration

Spyre became a native PyTorch device through PrivateUse1 registration, which provided real device identities for tensors and streams. The runtime patches programs at launch to substitute symbolic tensor addresses with real ones in three steps. In a demonstration on Granite 3.3 8B at batch size 1 and sequence length 1024, the team reported improvements of up to 2.4x faster decode times.

vLLM Integration

The integration also involved moving towards hardware-agnostic models within vLLM, which supports hardware-specific layers built on four design principles: compilable, extensible, isolated from hardware-specific paths, and portable. Early support for this has been validated with the Spyre out-of-tree plugin on various models including Gemma 4, Qwen3, Granite 4.2, and NVIDIA H100 GPUs.

Conclusion

The integration of Spyre as a first-class backend in PyTorch is a significant step forward, enabling more efficient and flexible AI workflows for enterprise teams using IBM’s systems.