Hugging Face adds GGUF support to Transformers for local inference
Hugging Face announced that the Transformers library can now load GGUF‑format checkpoints directly, letting users run quantized models on a laptop with the familiar frompretrained API. The feature reuses llama.cpp’s ggml kernels via the new kernels library and initially targets Apple Silicon, starting with the Qwen3.5 architecture. Users can pick a GGUF model from the Hub, specify the filename…
Key points
- Transformers now loads GGUF checkpoints via a gguffile argument in from_pretrained.
- Initial support focuses on Apple Silicon and Qwen3.5 models using ggml Metal kernels.
- Quantization options like Q4KM, Q5KM and Q6K let users fit larger models on laptop memory.
The integration aims to match llama.cpp’s performance on a MacBook Pro M2 Max (32 GB unified memory, macOS 26.6) using PyTorch 2.12.1 and kernels 0.17.0. Benchmarks on three checkpoints – a small dense model, a larger dense model, and a mixture‑of‑experts model – show token‑generation rates close to llama.cpp, despite the Transformers measurement including prefill. The update also includes two generation‑loop optimizations: dropping an unnecessary attention mask and deferring the stopping check, which reduce CPU‑GPU synchronization overhead.
GGUF models, which have been downloaded millions of times, support several quantization levels such as Q4KM, Q5KM and Q6K, allowing users to trade precision for a smaller memory footprint. The blog post thanks contributors including Arthur Zucker and Cyril Vallez, and invites the community to request additional model support.
The story so far
3 episodes →- Hugging Face adds GGUF support to Transformers for local inferencethis story
Transformers now runs llama.cpp quants
Hugging Face · 22 September 2026
Loading the full article…
This text was published by Hugging Face and written by Marc Sun. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
More in Generative AI & Models
All →- AI Safety Concerns Erupt Amid Recursive Self-Improvement Fears · 3 src
- Tencent launches Hy Image 3.5 preview, says it matches ByteDance's Seedream 5.0 Pro · 1 src
- Rumors of new AI models may surface next week · 1 src
- Austrian Academy of Science to release Apollo, first advanced Greek language model · 1 src
- Opinion: users complain about preachy ChatGPT, OpenAI tweaks GPT‑5.3 Instant · 1 src
Comments
via GitHub Discussions