{"version":1,"type":"story","url":"https://digestai.news/story/ggml-org-merges-cuda-patch-to-accelerate-low-bit-mmvq-in-llama-cpp","json":"https://digestai.news/story/ggml-org-merges-cuda-patch-to-accelerate-low-bit-mmvq-in-llama-cpp.json","markdown":"https://digestai.news/story/ggml-org-merges-cuda-patch-to-accelerate-low-bit-mmvq-in-llama-cpp.md","slug":"ggml-org-merges-cuda-patch-to-accelerate-low-bit-mmvq-in-llama-cpp","headline":"Ggml-org merges CUDA patch to accelerate low-bit MMVQ in llama.cpp","summary":"A pull request (#30517) to the ggml‑org/llama.cpp repository adds a CUDA patch that addresses a regression on GCN and introduces a fast path for several low‑bit quantization types. The change returns a single warp for Q3K (when ncolsdst is 2 or 3) and Q6K (when ncolsdst is 4), and adds a special case for RDNA2 GPUs that returns a single warp for Q2K when the column size equals 8.\\n\\nThe patch was authored by @am17an and approved by pwilkin. Review comments from thelittlefireman and IMbackK discuss further testing on AMD Instinct MI100 and Radeon RX 6800, noting that any slowdown under 5 % is considered noise. The contributors agree more work will be needed to handle other quantization types and to split table IDs for different AMD architectures.","keyPoints":["Patch adds a 1‑warp fast path for Q3K, Q6K, and Q2K low‑bit types on CUDA","Special handling for AMD RDNA2 GPUs (e.g., Radeon RX 6800) is included","Authors plan further benchmarking on AMD Instinct MI100 and other quant types"],"whyItMatters":"Improving low‑bit quantization speed in llama.cpp helps developers run large language models more efficiently on a wider range of GPUs.","category":{"slug":"models","name":"Generative AI & Models","url":"https://digestai.news/category/models"},"entities":{"companies":["AMD"],"models":[],"people":["am17an","pwilkin","thelittlefireman","IMbackK"]},"firstPublishedAt":"2026-10-14T00:00:00Z","updatedAt":"2026-10-14T00:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"github.com","title":"CUDA: faster MMVQ for low-bit types by am17an · Pull Request #30517 · ggml-org/llama.cpp","url":"https://github.com/ggml-org/llama.cpp/pull/30517","publishedAt":"2026-10-14T00:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[{"site":"Reddit","url":"https://www.reddit.com/r/LocalLLaMA/comments/1x8bqqs/cuda_faster_mmvq_for_lowbit_types_by_am17an_pull/","points":null}],"thread":{"title":"Ggml-org Optimizes Local AI Inference Performance","url":"https://digestai.news/thread/ggml-org-merges-vulkan-kernel-fusion-for-qwen4exp-s-scalesigmoidscale-chain","storyCount":4},"cite":{"text":"Digest AI, \"Ggml-org merges CUDA patch to accelerate low-bit MMVQ in llama.cpp\", 14 October 2026, https://digestai.news/story/ggml-org-merges-cuda-patch-to-accelerate-low-bit-mmvq-in-llama-cpp","publisher":"Digest AI","title":"Ggml-org merges CUDA patch to accelerate low-bit MMVQ in llama.cpp","datePublished":"2026-10-14T00:00:00Z","url":"https://digestai.news/story/ggml-org-merges-cuda-patch-to-accelerate-low-bit-mmvq-in-llama-cpp"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}