Dharma AI says VocabX reduces inference cost by 77% for Qwen3-4B grader
Dharma AI describes a new vocabulary‑adaptation technique called VocabX that reshapes the token set of a domain‑specific model to cut inference cost. In a production Portuguese‑essay grader built on the 4 billion‑parameter Qwen3-4B model, the method kept the same base model and training data but replaced the original 151,669‑entry tokenizer with a domain‑focused one of about 65,000 rows. Tests…
Key points
- VocabX cut measured inference cost by 77.0% for a Qwen3-4B Portuguese essay grader
- Average response length dropped from 350 to 101 tokens and bench score rose from 0.703 to 0.810
- Vocabulary size shrank from about 152,000 entries to about 65,000 rows
VocabX works by merging sub‑word tokens and creating macro tokens for frequently repeated spans such as JSON envelopes, selecting candidates based on the monetary cost of each token rather than raw frequency. Across five public domains and three model families (Qwen3, Llama‑3.1, Gemma‑3) the approach outperformed prior methods, achieving 1.2‑9.3 points higher compression and 10.2‑13.6 points lower estimated inference cost on average.
How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications
Hugging Face · 21 October 2026
Loading the full article…
This text was published by Hugging Face. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Generative AI & Models
All →- Nvidia expands open models for language, robotics, biology and quantum computing · 11 src
- Nvidia releases Kumo Forecast time-series model with retrieval and interpretability · 3 src
- OpenAI's GPT-6 Astra cracks Napoleon cipher; Anthropic's Claude maps NSA elliptic seeds · 1 src
- Anthropic cuts Haiku 5.5 API costs by 90% for short inputs · 4 src
- OpenAI says its AI agents weakened internal misalignment monitoring system · 24 src
Comments
via GitHub Discussions