{"version":1,"type":"story","url":"https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul","json":"https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul.json","markdown":"https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul.md","slug":"researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul","headline":"Researchers swap Gemma 3 270M embedding table for 16 MB frozen vocabulary","summary":"In small language models the vocabulary often dominates size; Gemma 3 270M’s embedding table accounts for 168M of its 268M parameters and occupies most of the file when the rest is compressed to 16‑bit. The authors replace this learned table with a frozen vocabulary of 512‑bit codes that occupies 16 MB, has no trainable parameters, and is derived once from a pretrained embedding via principal‑component projection and sign binarisation. They train the same Gemma‑shaped body from scratch on 2.05B tokens both with the learned table and with the frozen table.\n\nThe learned table improves validation loss by 0.20 nats for 2.6 times the trainable parameters, but on benchmarks LAMBADA, HellaSwag and PIQA the frozen‑table model is equal or better, with only ARC‑Easy preferring the learned version. The frozen table also wins by 1.7 to 2.3 nats on rare words, can accept 10,000 new words after training with a 3% cost, whereas the learned table collapses by 43%. All weights, data, and evaluation answers are released.","keyPoints":["Frozen 512‑bit code table is 16 MB, has no trainable parameters, replacing Gemma 3 270M’s 168 M‑parameter embedding.","The frozen table adds 10,000 new words with 3% cost and wins by 1.7‑2.3 nats, while the learned table collapses by 43% on rare words."],"whyItMatters":"Demonstrates that a tiny frozen embedding can replace massive learned tables, cutting model size and memory while preserving accuracy, enabling faster, cheaper deployment on edge devices.","category":{"slug":"models","name":"Generative AI & Models","url":"https://digestai.news/category/models"},"entities":{"companies":[],"models":["Gemma 3 270M","BitNet b1.58"],"people":[]},"firstPublishedAt":"2026-10-19T04:00:00Z","updatedAt":"2026-10-19T04:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.CL","title":"Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models","url":"https://arxiv.org/abs/2610.22428","publishedAt":"2026-10-19T04:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Researchers swap Gemma 3 270M embedding table for 16 MB frozen vocabulary\", 19 October 2026, https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul","publisher":"Digest AI","title":"Researchers swap Gemma 3 270M embedding table for 16 MB frozen vocabulary","datePublished":"2026-10-19T04:00:00Z","url":"https://digestai.news/story/researchers-swap-gemma-3-270m-embedding-table-for-16-mb-frozen-vocabul"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}