# MarkTechPost explains GGUF, GPTQ, AWQ, EXL2, EXL3 model formats

Digest AI · Generative AI & Models · published 2026-09-19T04:04:06Z

Canonical: https://digestai.news/story/marktechpost-explains-gguf-gptq-awq-exl2-exl3-model-formats

## Summary

The article separates two layers of model handling: containers that store tensors on disk and quantization methods that compress weights. Containers mentioned are safetensors, GGUF and the older PyTorch pickle files (.bin/.pt). The latter can execute arbitrary code, so untrusted checkpoints pose a security risk, while safetensors avoid that risk by using a simple JSON header and raw buffers. Quantization methods listed include GPTQ, AWQ, bitsandbytes NF4, and llama.cpp K‑quants and I‑quants.

GGUF, a binary format created by Georgi Gerganov, was introduced on Aug 21 2023 as the successor to GGML. It adds typed key‑value metadata, allowing a single file to hold weights, tokenizer, special tokens and a chat template. The spec shows how a Q4KM file averages about 4.5 bits per weight after accounting for scales and super‑blocks. The format is supported by llama.cpp, LM Studio, GPT4All, Ollama and, experimentally, vLLM.

Calibration for an 8B model takes roughly 20 minutes on 1 A100. AWQ, from Song Han’s MIT group, protects the top 1 % of salient weights, runs more than 3× faster than FP16, and calibrates an 8B model in about 10 minutes on 1 A100. EXL2 (ExLlamaV2) supports 2‑8‑bit mixed quantization with automatic bitrate selection; EXL3 (ExLlamaV3) converts a 70B model in a few hours on an RTX 4090‑class GPU and runs Llama‑3.1‑70B at 1.6 bits per weight under 16 GB VRAM. Bitsandbytes NF4 enables fine‑tuning a 65B model on a single 48 GB GPU, and MLX provides Apple‑Silicon‑specific safetensors with its own quantized weights.

## Key points

- GGUF is a binary container introduced Aug 21 2023, replacing GGML and storing weights, tokenizer, and chat template in one file.
- GPTQ quantizes 175B‑parameter models to 3‑4 bits per weight in ~4 GPU hours, with negligible accuracy loss per the authors.
- EXL3 can convert a 70B model in a few hours on RTX 4090‑class GPU and runs at 1.6 bits per weight for Llama‑3.1‑70B under 16 GB VRAM.

## Why it matters

Understanding containers and quantization methods lets developers pick formats that fit storage limits, hardware capabilities, and performance goals, reducing deployment cost across CPUs, GPUs and Apple Silicon.

## Sources

1. [GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained](https://marktechpost.com/2026/09/18/gguf-vs-gptq-vs-awq-vs-exl2-llm-model-formats-explained-2026) (MarkTechPost, 2026-09-19)

Part of the developing story: [Qwen Model Optimization and Format Clarification](https://digestai.news/thread/qwen3-8-27b-gguf-updated-with-new-tensor-layout) (2 stories)

## Cite

Digest AI, "MarkTechPost explains GGUF, GPTQ, AWQ, EXL2, EXL3 model formats", 19 September 2026, https://digestai.news/story/marktechpost-explains-gguf-gptq-awq-exl2-exl3-model-formats

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/marktechpost-explains-gguf-gptq-awq-exl2-exl3-model-formats.json