{"version":1,"type":"story","url":"https://digestai.news/story/sfnx-runs-qwen3-8-flash-next-at-382-tokens-s-using-three-used-v100-gpu","json":"https://digestai.news/story/sfnx-runs-qwen3-8-flash-next-at-382-tokens-s-using-three-used-v100-gpu.json","markdown":"https://digestai.news/story/sfnx-runs-qwen3-8-flash-next-at-382-tokens-s-using-three-used-v100-gpu.md","slug":"sfnx-runs-qwen3-8-flash-next-at-382-tokens-s-using-three-used-v100-gpu","headline":"sFNX runs Qwen3.8-Flash-Next at 382 tokens/s using three used V100 GPUs","summary":"sFNX built an engine that runs the Qwen3.8-Flash-Next model at 382 tokens per second on a code‑edit benchmark. The setup uses three used Tesla V100‑32GB GPUs that cost about $1,800 in total and are connected via PCIe Gen3 without NVLink.\n\nThe engine splits every transformer layer across the three cards, letting each request read weights from all cards simultaneously. Six concurrent conversations are kept, each with the full 262,144‑token context, and the system also serves OpenAI‑style chat, Anthropic messages, tool calls and a picture front‑end. Quantizations UD‑Q4KXL and Q80 run in production, and the code relies on llama.cpp, CUDA 12.9 and custom NCCL‑like summation inside a CUDA graph.\n\nBenchmarks show a range of runs, with the headline 382 tokens/s being the best of three greedy code‑edit runs. The configuration is documented in the repo’s docs, and the engine starts in about 80 seconds, exposing an API on port 8080.","keyPoints":["sFNX runs Qwen3.8-Flash-Next at 382 tokens per second on three V100‑32GB GPUs costing ~$1,800.","The engine splits every model layer across three cards, avoiding NVLink and using custom CUDA‑graph sums.","Six concurrent 262,144‑token conversations are served, including OpenAI chat, Anthropic messages and picture front‑end."],"whyItMatters":"Shows that large‑scale LLMs can be deployed on inexpensive, older GPUs, lowering entry barriers for developers and small teams seeking high‑throughput inference.","category":{"slug":"models","name":"Generative AI & Models","url":"https://digestai.news/category/models"},"entities":{"companies":["sFNX","OpenAI","Anthropic","Unsloth","NVIDIA"],"models":["Qwen3.8-Flash-Next","UD-Q4KXL","Q80"],"people":[]},"firstPublishedAt":"2026-10-26T21:11:25Z","updatedAt":"2026-10-26T21:11:25Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"github.com","title":"I built an engine that runs Qwen-Flash-Next at nearly 400t/s on cheap(ish) hardware. 3x V100 32GB in an old R740. Originally forked from strata but the engine guts all built for the old Volta generation.","url":"https://github.com/johnfink8/sfnx","publishedAt":"2026-10-26T21:11:25Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[{"site":"Reddit","url":"https://www.reddit.com/r/LocalLLaMA/comments/1xg9cr1/i_built_an_engine_that_runs_qwenflashnext_at/","points":null}],"thread":{"title":"Qwen3.8 Flash Next Optimization Sprint","url":"https://digestai.news/thread/inco-ai-releases-dflash-2-draft-model-for-qwen3-8-flash-next","storyCount":4},"cite":{"text":"Digest AI, \"sFNX runs Qwen3.8-Flash-Next at 382 tokens/s using three used V100 GPUs\", 26 October 2026, https://digestai.news/story/sfnx-runs-qwen3-8-flash-next-at-382-tokens-s-using-three-used-v100-gpu","publisher":"Digest AI","title":"sFNX runs Qwen3.8-Flash-Next at 382 tokens/s using three used V100 GPUs","datePublished":"2026-10-26T21:11:25Z","url":"https://digestai.news/story/sfnx-runs-qwen3-8-flash-next-at-382-tokens-s-using-three-used-v100-gpu"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}