# Researchers swap Gemma 3 270M embedding table for 16 MB frozen vocabulary

Digest AI · Generative AI & Models · published 2026-10-19T04:00:00Z

Canonical: https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul

## Summary

In small language models the vocabulary often dominates size; Gemma 3 270M’s embedding table accounts for 168M of its 268M parameters and occupies most of the file when the rest is compressed to 16‑bit. The authors replace this learned table with a frozen vocabulary of 512‑bit codes that occupies 16 MB, has no trainable parameters, and is derived once from a pretrained embedding via principal‑component projection and sign binarisation. They train the same Gemma‑shaped body from scratch on 2.05B tokens both with the learned table and with the frozen table.

The learned table improves validation loss by 0.20 nats for 2.6 times the trainable parameters, but on benchmarks LAMBADA, HellaSwag and PIQA the frozen‑table model is equal or better, with only ARC‑Easy preferring the learned version. The frozen table also wins by 1.7 to 2.3 nats on rare words, can accept 10,000 new words after training with a 3% cost, whereas the learned table collapses by 43%. All weights, data, and evaluation answers are released.

## Key points

- Frozen 512‑bit code table is 16 MB, has no trainable parameters, replacing Gemma 3 270M’s 168 M‑parameter embedding.
- The frozen table adds 10,000 new words with 3% cost and wins by 1.7‑2.3 nats, while the learned table collapses by 43% on rare words.

## Why it matters

Demonstrates that a tiny frozen embedding can replace massive learned tables, cutting model size and memory while preserving accuracy, enabling faster, cheaper deployment on edge devices.

## Sources

1. [Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models](https://arxiv.org/abs/2610.22428) (arXiv cs.CL, 2026-10-19, primary source)

## Cite

Digest AI, "Researchers swap Gemma 3 270M embedding table for 16 MB frozen vocabulary", 19 October 2026, https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul.json