{"version":1,"type":"story","url":"https://digestai.news/story/ggml-org-updates-llama-cpp-with-flash-attention-tuning-for-amd-gfx1201","json":"https://digestai.news/story/ggml-org-updates-llama-cpp-with-flash-attention-tuning-for-amd-gfx1201.json","markdown":"https://digestai.news/story/ggml-org-updates-llama-cpp-with-flash-attention-tuning-for-amd-gfx1201.md","slug":"ggml-org-updates-llama-cpp-with-flash-attention-tuning-for-amd-gfx1201","headline":"CachyLLama: AMD-optimized llama.cpp fork for local LLM inference on APUs","summary":"CachyLLama is a performance-focused fork of llama.cpp designed specifically for running local large language models on AMD APU hardware, including integrated GPUs and handheld devices. Unlike upstream llama.cpp, which prioritizes raw model execution, CachyLLama optimizes for end-to-end agentic throughput on shared-memory systems. It introduces several key innovations, including a persistent KV cache that survives server restarts and power cycles, and a three-tier architecture (hot, warm, cold) that automatically manages conversation state across RAM and disk.\n\nThe project features advanced MoE expert residency, allowing models larger than physical memory to run at near-RAM speeds by keeping only active expert weights in memory. Benchmarks show significant speedups on AMD Radeon hardware, such as a 41.1% increase in decode speed for DeepSeek-V4 models. The fork also includes custom Vulkan optimizations, such as the Lightning Indexer and quantized-KV flash-attention improvements, tailored for RDNA3 and Strix Halo architectures.\n\nCachyLLama supports a wide range of model architectures, including Qwen3.5/3.6, Gemma 4, and GLM-4.7, with specific support for hybrid MoE/SSM models. It is released under the MIT license, with the parent project using GPL-3.0. The ecosystem includes CLIO, an agentic client optimized for CachyLLama's persistent cache features, and provides detailed documentation on expert activation tracking and user isolation mechanisms.","keyPoints":["CachyLLama optimizes llama.cpp for AMD APUs, achieving up to 41% decode speedup on DeepSeek-V4.","Features persistent KV cache and MoE expert residency to run large models on limited RAM.","Includes custom Vulkan shaders like Lightning Indexer and quantized-KV flash-attention for RDNA3 hardware."],"whyItMatters":"This fork lowers the barrier for running high-performance local LLMs on consumer AMD hardware, enabling agentic workflows on devices with limited unified memory and extending the reach of open-source AI inference.","category":{"slug":"hardware","name":"Hardware & Compute","url":"https://digestai.news/category/hardware"},"entities":{"companies":["AMD"],"models":["llama.cpp","DeepSeek-V4","Qwen3.6","Gemma 4","GLM-4.7"],"people":[]},"firstPublishedAt":"2026-09-11T09:27:58Z","updatedAt":"2026-09-14T06:12:42Z","sourceCount":3,"hasPrimarySource":true,"sources":[{"outlet":"github.com","title":"CachyLlama archived?","url":"https://github.com/fewtarius/CachyLLama-archived","publishedAt":"2026-09-14T06:12:42Z","type":"primary","primary":true,"lead":true},{"outlet":"github.com","title":"ggml-cuda: hip: add missing AMD GCN MMQ config by thelittlefireman · Pull Request #27841 · ggml-org/llama.cpp - PP improvements for RDNA2(MI50, MI60)","url":"https://github.com/ggml-org/llama.cpp/pull/27841","publishedAt":"2026-09-12T09:53:22Z","type":"primary","primary":true,"lead":false},{"outlet":"github.com","title":"CUDA/HIP: Flash Attention tuning (gfx1201) by pwilkin · Pull Request #28102 · ggml-org/llama.cpp","url":"https://github.com/ggml-org/llama.cpp/pull/28102","publishedAt":"2026-09-11T09:27:58Z","type":"primary","primary":true,"lead":false}],"sourceNotes":null,"discussions":[{"site":"Reddit","url":"https://www.reddit.com/r/LocalLLaMA/comments/1wfvcwa/cachyllama_archived/","points":null},{"site":"Reddit","url":"https://www.reddit.com/r/LocalLLaMA/comments/1we8elm/ggmlcuda_hip_add_missing_amd_gcn_mmq_config_by/","points":null},{"site":"Reddit","url":"https://www.reddit.com/r/LocalLLaMA/comments/1wdbal8/cudahip_flash_attention_tuning_gfx1201_by_pwilkin/","points":null}],"thread":{"title":"AMD GPUs Finally Run CUDA Workloads","url":"https://digestai.news/thread/ggml-org-updates-llama-cpp-with-flash-attention-tuning-for-amd-gfx1201","storyCount":3},"cite":{"text":"Digest AI, \"CachyLLama: AMD-optimized llama.cpp fork for local LLM inference on APUs\", 11 September 2026, https://digestai.news/story/ggml-org-updates-llama-cpp-with-flash-attention-tuning-for-amd-gfx1201","publisher":"Digest AI","title":"CachyLLama: AMD-optimized llama.cpp fork for local LLM inference on APUs","datePublished":"2026-09-11T09:27:58Z","url":"https://digestai.news/story/ggml-org-updates-llama-cpp-with-flash-attention-tuning-for-amd-gfx1201"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}