Bitcos layout cuts ternary LLM storage below 1.58 bits per weight
Researchers have introduced BITCOS, a distribution‑adaptive storage format for ternary large language models that leverages the high prevalence of zero weights. By encoding a dense presence bitmap together with a compact sign vector, BITCOS reduces the effective bit‑width to 2 – z bits per weight, where z is the zero density. In tests on 29 ternary LLMs, the method outperformed the traditional…
Key points
- BITCOS reduces ternary LLM storage to as low as 1.485 bits per weight.
- Achieves up to 1.28× faster matrix‑vector multiplication versus five‑trit packing.
- Inference throughput improves up to 1.27× on Intel Xe2 GPUs.
The authors also optimized unpacking routines for modern CPUs and GPUs, including AVX‑512, AVX2, and Intel Xe2 GPUs. Benchmarks show up to a 1.28× speedup in matrix‑vector multiplication and end‑to‑end inference gains of 1.18× on CPUs and 1.27× on GPUs across five hardware platforms. These improvements make ternary LLMs more storage‑efficient and faster to run, potentially lowering deployment costs for edge and server environments.
Breaking the 1.58-bit Barrier for Ternary LLMs
arxiv.org · 16 September 2026
Loading the full article…
This text was published by arxiv.org. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Hacker News discussion · 67 pointsnews.ycombinator.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- PyTorch tutorial shows how to cut drone data usage by 94% · 1 src
- Article outlines five practices for building robust Python AI libraries · 1 src
- Vector databases explain how AI retrieves similar data at scale · 1 src
- PhAI Labs introduces JEPA-Anything framework for cross-domain world models · 2 src
- Researchers introduce VIMA for finding disease patterns in tissue data · 2 src
Comments
via GitHub Discussions