DigestAI news desk
hot Microsoft AI CEO warns Anthropic's model rights approach risks alignment failures new OpenAI unveils framework for public disclosure of AI misalignment incidents hot Anthropic merges Cowork and chat into unified 'One Claude' platform new Apple plans 2029 AI server launch using M8 Ultra and Nvidia NVLink tech hot BITCOS layout cuts ternary LLM storage below 1.58 bits per weight new OpenAI CEO Altman to Attend Trump-Xi State Dinner hot TypeSafe AI unveils Jev, a frontier model up to 400× cheaper and 200× faster new Google launches Home MCP for AI agent smart home control hot Apple Unveils macOS 27 Golden Gate with Built‑In Generative AI and Intel‑Free Design new OpenAI Postpones IPO Amid Safety Concerns, $1.2T Valuation Talk hot OpenAI launches Sponsored Agents and AI ad tools in ChatGPT new AI Model Pace Concerns Trigger Nvidia, AMD, Intel Stock Selloff hot Google DeepMind launches Gemini 3.8 Live and 3.8 Live Extended Thinking models new OpenAI investors propose $1.2 trillion valuation for new funding round hot Langdock Reorganises Parent Company into German SE to Reduce US Dependence new Novo Nordisk partners with Anthropic to accelerate drug discovery using Claude
Research updated 2 min read

BITCOS layout cuts ternary LLM storage below 1.58 bits per weight

Researchers have introduced BITCOS, a distribution‑adaptive storage format for ternary large language models that leverages the high prevalence of zero weights. By encoding a dense presence bitmap together with a compact sign vector, BITCOS reduces the effective bit‑width to 2 – z bits per weight, where z is the zero density. In tests on 29 ternary LLMs, the method outperformed the traditional…

1 source primary source HN 67

Key points

  • BITCOS reduces ternary LLM storage to as low as 1.485 bits per weight.
  • Achieves up to 1.28× faster matrix‑vector multiplication versus five‑trit packing.
  • Inference throughput improves up to 1.27× on Intel Xe2 GPUs.

The authors also optimized unpacking routines for modern CPUs and GPUs, including AVX‑512, AVX2, and Intel Xe2 GPUs. Benchmarks show up to a 1.28× speedup in matrix‑vector multiplication and end‑to‑end inference gains of 1.18× on CPUs and 1.27× on GPUs across five hardware platforms. These improvements make ternary LLMs more storage‑efficient and faster to run, potentially lowering deployment costs for edge and server environments.

Full story from arxiv.org · via Hacker News primary source Open source ↗

Breaking the 1.58-bit Barrier for Ternary LLMs

arxiv.org · 16 September 2026

Computer Science > Artificial Intelligence

Title:Breaking the 1.58-bit Barrier for Ternary LLMs

View PDF HTML (experimental) Abstract:Ternary Large Language Models (LLM) store every weight as one of three symbols ${-1,0,+1}$, so the cost of a ternary model is conventionally referenced to the information-theoretic $\log_2 3 \approx 1.585$ bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to $1.625$ bits per weight. This effective storage bit-width treats the three symbols ${-1,0,+1}$ as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to $51.5%$ of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs $2 - z$ bits per weight element given a zero density $z$ in the model's weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches $1.485$ bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to $1.28\times$. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to $1.18\times$ on CPUs and $1.27\times$ on GPUs.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

This text was published by arxiv.org . It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗

Coverage and discussion

1 source
Topics · follow one to build your own front page
Intel

The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.

Comments

via GitHub Discussions

More in Research

All →

Related stories