CachyLLama: AMD-optimized llama.cpp fork for local LLM inference on APUs

The project features advanced MoE expert residency, allowing models larger than physical memory to run at near-RAM speeds by keeping only active expert weights in memory. Benchmarks show significant speedups on AMD Radeon hardware, such as a 41.1% increase in decode speed for DeepSeek-V4 models. The fork also includes custom Vulkan optimizations, such as the Lightning Indexer and quantized-KV flash-attention improvements, tailored for RDNA3 and Strix Halo architectures.

CachyLLama supports a wide range of model architectures, including Qwen3.5/3.6, Gemma 4, and GLM-4.7, with specific support for hybrid MoE/SSM models. It is released under the MIT license, with the parent project using GPL-3.0. The ecosystem includes CLIO, an agentic client optimized for CachyLLama's persistent cache features, and provides detailed documentation on expert activation tracking and user isolation mechanisms.