Researchers swap Gemma 3 270M embedding table for 16 MB frozen vocabulary
In small language models the vocabulary often dominates size; Gemma 3 270M’s embedding table accounts for 168M of its 268M parameters and occupies most of the file when the rest is compressed to 16‑bit. The authors replace this learned table with a frozen vocabulary of 512‑bit codes that occupies 16 MB, has no trainable parameters, and is derived once from a pretrained embedding via…
Key points
- Frozen 512‑bit code table is 16 MB, has no trainable parameters, replacing Gemma 3 270M’s 168 M‑parameter embedding.
- The frozen table adds 10,000 new words with 3% cost and wins by 1.7‑2.3 nats, while the learned table collapses by 43% on rare words.
The learned table improves validation loss by 0.20 nats for 2.6 times the trainable parameters, but on benchmarks LAMBADA, HellaSwag and PIQA the frozen‑table model is equal or better, with only ARC‑Easy preferring the learned version. The frozen table also wins by 1.7 to 2.3 nats on rare words, can accept 10,000 new words after training with a 3% cost, whereas the learned table collapses by 43%. All weights, data, and evaluation answers are released.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Generative AI & Models
All →- Anthropic’s Claude Opus 5.5 tops Arena AI model rankings · 3 src
- DeepSeek releases V4.1-Flash-1017 update with lower cost and higher DeepSWE score · 20 src
- KillSwitch language challenges LLMs, Claude Opus scores highest · 1 src
- The Truth Behind 'Apple and Meta Team Up on AI': It's Not Siri, It's Mac Studio | A Manager's Breakdown of Confirmed Facts, Reports, Unannounced Details, and Im · 1 src
- OpenAI makes Prompt-to-Voice generally available for developers · 3 src
Comments
via GitHub Discussions