GitHub - giveen/ninfer-ext: ninfer-ext — He built the product, we are building the weapon.

The project fits the large model into consumer hardware by keeping routed experts in pinned host RAM—requiring around 128 GB of system memory and 127 GB of disk space—while streaming active layers over PCIe into a device expert cache. The fork also improves speculative decoding performance, reporting 13% to 48% speedups using DFlash2 on Qwen3.8-27B groupwise-int across one to four concurrent requests compared to stock ninfer.

Additional updates target agent workflows, including out-of-memory recovery, context-cache salvage for aborted requests, and OpenAI- and Anthropic-compatible local HTTP endpoints. The developer notes that the tool only supports the RTX 5090 architecture and ties upstream performance on certain models like Qwen3.6-35B-A3B.