sFNX runs Qwen3.8-Flash-Next at 382 tokens/s using three used V100 GPUs
sFNX built an engine that runs the Qwen3.8-Flash-Next model at 382 tokens per second on a code‑edit benchmark. The setup uses three used Tesla V100‑32GB GPUs that cost about $1,800 in total and are connected via PCIe Gen3 without NVLink.
Key points
- sFNX runs Qwen3.8-Flash-Next at 382 tokens per second on three V100‑32GB GPUs costing ~$1,800.
- The engine splits every model layer across three cards, avoiding NVLink and using custom CUDA‑graph sums.
- Six concurrent 262,144‑token conversations are served, including OpenAI chat, Anthropic messages and picture front‑end.
The engine splits every transformer layer across the three cards, letting each request read weights from all cards simultaneously. Six concurrent conversations are kept, each with the full 262,144‑token context, and the system also serves OpenAI‑style chat, Anthropic messages, tool calls and a picture front‑end. Quantizations UD‑Q4KXL and Q80 run in production, and the code relies on llama.cpp, CUDA 12.9 and custom NCCL‑like summation inside a CUDA graph.
Benchmarks show a range of runs, with the headline 382 tokens/s being the best of three greedy code‑edit runs. The configuration is documented in the repo’s docs, and the engine starts in about 80 seconds, exposing an API on port 8080.
The story so far
4 episodes →- sFNX runs Qwen3.8-Flash-Next at 382 tokens/s using three used V100 GPUsthis story
I built an engine that runs Qwen-Flash-Next at nearly 400t/s on cheap(ish) hardware. 3x V100 32GB in an old R740. Originally forked from strata but the engine guts all built for the old Volta generation.
github.com · 26 October 2026Loading the full article…
This text was published by github.com. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Reddit discussionreddit.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Generative AI & Models
All →- Moonshot’s Kimi K3.1 Pro Takes Aim at US AI Leaders · 3 src
- OpenAI announces GPT-6.1 Sol with claimed 86.6% to 97.9% COBOL migration accuracy · 8 src
- CodeRabbit uses GPT‑6.1 Sol in security agent · 1 src
- Anthropic reports 3.6x jump in token volume after Claude Opus 5.5 launch · 3 src
- AI models may struggle with clear communication, study finds · 2 src
Comments
via GitHub Discussions