Open-Source AI Surge Local Models Faster Inference
The saga chronicles the rapid expansion of open‑source AI tools that can run locally, from ChatGPT alternatives to multimodal models and performance‑boosting tweaks. Today, developers can leverage Vulkan‑accelerated llama.cpp on AMD GPUs, achieving a 12.9% speed increase and accessing new Q4_0 ROCmFP4 models.
-
gufo-Qwen3.6-35B-A3B-Q6dense - 3095tok/s prefill; 190 tok/s decode on Strix Halo
The Gufo project posted benchmark results for its vertical local inference engine built for AMD Strix Halo hardware (Ryzen AI MAX+ 395 with Radeon 8060S, up to 128 GiB unified memory). The engine…
2 sources primary source -
[For 780M] Vulkan inference 12.9% faster: MMVQ tuning for AMD 780M/Strix in llama.cpp, Q4_0 ROCmFP4 models arriving one after another
By setting load‑mode to auto, the build now skips memory‑mapped I/O on iGPUs, preventing the model from being loaded twice and halving usable memory. New ROCmFP4 quantized models for Strix/Halo are…
1 source -
Liquid AI releases LFM2.5-VL-DSpark for faster vision-language model inference
Liquid AI announced LFM2.5-VL-DSpark, a draft model for its LFM2.5-VL-3B vision-language model. The model uses speculative decoding to speed up inference by up to 3.13x on-device and 2.66x on H100…
2 sources primary source -
Open-source panel lets users run llama.cpp, stable-diffusion.cpp and audio.cpp locally
A community‑maintained control panel provides a single‑page web UI for running local AI inference on a personal GPU box. It supports llama.cpp for text and vision chat, stable-diffusion.cpp for…
1 source primary source