Jev may offer cheaper, faster clinical AI judgments than frontier LLM judges
Jev, a decision language model that returns structured judgments, was evaluated against a suite of frontier and open‑weight large language models for assessing clinical AI outputs. In a test of 118 pediatric answers, Jev’s agreement with physicians’ ordinal quality ratings (Gwet’s AC2 0.50; expected score 0.53) matched or exceeded that of seven of twelve LLM judges, including Claude Opus 5.5…
Key points
- Jev matched or exceeded physician agreement of seven of twelve LLM judges, including Claude Opus 5.5 (0.49) and GPT‑6 (0.38).
- On HealthBench Professional, Jev matched the reference judge GPT‑5.4 on 91% of criteria, similar to other judges (91‑93%).
- Frontier LLM judges cost 73‑294× more and took 7‑57× longer than Jev.
On physician‑graded binary HealthBench criteria, Jev matched or outperformed four of six LLM judges, and on MedHELM’s open‑ended tasks its agreement with a three‑LLM jury approached the jurors’ internal consistency. For the HealthBench Professional benchmark, Jev matched the reference judge GPT‑5.4 on 91 % of criteria, comparable to other judges that scored 91‑93 %. The study notes that frontier judges cost 73‑294 times more and took 7‑57 times longer to run than Jev, suggesting Jev may enable efficient, large‑scale clinical AI evaluation.
Model pages: Jev → · Claude Opus 5.5 → · GPT-6 →
The story so far
5 episodes →- Jev may offer cheaper, faster clinical AI judgments than frontier LLM judgesthis story
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- PHP patches critical RCE flaws discovered with GPT-5.6 Sol, affecting up to 70% of websites · 9 src
- OpenAI and Google AI models discovered critical PHP flaw affecting millions of websites · 8 src
- Researchers release madartb, a parallel Multi-Dialect Arabic dependency treebank · 1 src
- Researchers decode 26 legal factors in three open LLMs · 1 src
- Researchers test how reinforcement learning teaches AI agents to enforce market collusion · 1 src
Comments
via GitHub Discussions