Researchers test Typed-Decision models on Indonesian regional languages
A new study evaluates typed-decision models—fast, low-cost systems that return labels and probabilities in one step—on Indonesian regional languages. The paper, posted on arXiv, compares the commercial Jev model and the open-source Laya-multilingual model using two datasets: NusaX sentiment (12 languages) and NusaTranslation sentiment and emotion (7 languages).
Key points
- Typed-decision models like **Jev** and **Laya-multilingual** tested on 12+ Indonesian regional languages using **NusaX** and **NusaTranslation** datasets
- **Jev**’s accuracy rises by up to **37.2 points** with Indonesian translations for distant languages, but self-error ranking degrades
- A hybrid cascade (routing low-confidence outputs to **IndoBERT**) matches LLM performance at **24–77% lower cost**
The findings show Jev performs better on sentences closer to Indonesian but struggles with distant languages, where accuracy improves by up to 37.2 points when given human Indonesian translations. While its calibration error remains stable, its ability to self-rank errors weakens. Retraining classifiers requires more than 500 labels per language to match Jev’s zero-shot performance in nine of ten NusaX languages. A hybrid approach—routing Jev’s least confident decisions (10–30%) to a local IndoBERT classifier—matches or exceeds the best system at 24–77% lower API cost than routing to an LLM. Five other commercial LLMs (three Claude and three GPT models) outperform Jev by 4–14 points but cost hundreds of times more.
Model page: Jev →
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Asset learns to compress speech encoders · 1 src
- Researchers find Qwen3’s reasoning gains may stem from ‘thinking leakage’ · 1 src
- Researchers test AI agents improving LLM training strategies with up to 13.34% gains · 1 src
- Researchers model how AI and humans time new words in text · 1 src
- Researchers propose Next-Concept Modeling for faster text generation · 1 src
Comments
via GitHub Discussions