# Dharma AI says VocabX reduces inference cost by 77% for Qwen3-4B grader

Digest AI · Generative AI & Models · published 2026-10-21T17:35:36Z

Canonical: https://digestai.news/story/dharma-ai-says-vocabx-reduces-inference-cost-by-77-for-qwen3-4b-grader

## Summary

Dharma AI describes a new vocabulary‑adaptation technique called VocabX that reshapes the token set of a domain‑specific model to cut inference cost. In a production Portuguese‑essay grader built on the 4 billion‑parameter Qwen3-4B model, the method kept the same base model and training data but replaced the original 151,669‑entry tokenizer with a domain‑focused one of about 65,000 rows. Tests on an RTX 5090 GPU running 500 requests through the vLLM server showed the average response length fell from 350 to 101 tokens and measured runtime cost dropped 77.0%, while the grading benchmark score rose from 0.703 to 0.810. The paper, still under review, notes the numbers may change before publication.

VocabX works by merging sub‑word tokens and creating macro tokens for frequently repeated spans such as JSON envelopes, selecting candidates based on the monetary cost of each token rather than raw frequency. Across five public domains and three model families (Qwen3, Llama‑3.1, Gemma‑3) the approach outperformed prior methods, achieving 1.2‑9.3 points higher compression and 10.2‑13.6 points lower estimated inference cost on average.

## Key points

- VocabX cut measured inference cost by 77.0% for a Qwen3-4B Portuguese essay grader
- Average response length dropped from 350 to 101 tokens and bench score rose from 0.703 to 0.810
- Vocabulary size shrank from about 152,000 entries to about 65,000 rows

## Why it matters

Reducing token‑level cost and model size can make small LLMs far cheaper to run in production, expanding feasible AI services.

## Sources

1. [How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications](https://huggingface.co/blog/Dharma-AI/vocabx-vocabulary-specialization) (Hugging Face, 2026-10-21, primary source)

## Cite

Digest AI, "Dharma AI says VocabX reduces inference cost by 77% for Qwen3-4B grader", 21 October 2026, https://digestai.news/story/dharma-ai-says-vocabx-reduces-inference-cost-by-77-for-qwen3-4b-grader

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/dharma-ai-says-vocabx-reduces-inference-cost-by-77-for-qwen3-4b-grader.json