Researchers release MedJev, a zero-shot medical AI model for laptops
Researchers introduced MedJev, an open-source System One model that answers medical questions in one pass without text generation. It runs locally on a laptop and matches GPT-4 performance on benchmarks like MedQA (72.5% accuracy), USMLE (75.4%), and NEJM (80.1%), while keeping computational costs low.
Key points
- MedJev achieves 72.5% accuracy on MedQA, 75.4% on USMLE, and 80.1% on NEJM, nearing GPT-4 levels
- Runs locally on a laptop with low computational cost, unlike large cloud-based models
- Researchers propose a recalibration method reducing miscalibration error from 0.286 to 0.007
MedJev outperforms zero-shot LLMs and even XGBoost on In-Hospital Mortality (AUROC 0.926) but lags behind supervised models on tasks like ED Triage Admission. The team also developed a recalibration method using just 50–200 labeled examples to improve prediction accuracy. The code and evaluation framework are open for further research.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers propose synthetic document finetuning to detect alignment-faking LLMs · 3 src
- Researchers introduce LOTUS method to detect LLM hallucinations via low-rank decomposition · 1 src
- Researchers find Instruction tuning sharpens Number-Choice heuristics in language models · 1 src
- Researchers release FinLLM-KG framework for financial sentiment analysis using knowledge graphs · 1 src
- Researchers introduce ReCoPII framework for PII masking datasets · 1 src
Comments
via GitHub Discussions