Protege launches independent healthcare AI benchmark tests
Protege announced a new service that runs independent, double‑blind evaluations of healthcare AI on patient records the models have never seen. Its first benchmark, called Disease Odyssey, showed that leading models identified conditions such as sleep apnea and anemia in only 53%–69% of cases when working from longitudinal histories. The launch also featured tests of Xlue AI’s HealthSAW model…
Key points
- Protege's Disease Odyssey benchmark found leading models identified conditions in only 53%–69% of cases.
- Standard Model Biomedicine and Xlue AI's HealthSAW claimed 36–170× lower prediction cost and outperformed on most tests.
- Protege uses double‑blind, privacy‑protected patient records to evaluate AI, aiming to give hospitals and developers clearer performance data.
Protege’s platform uses privacy‑protected records, clinician‑verified answers, and versioned tasks to keep test material separate from development data, aiming to prevent benchmark over‑fitting. Co‑founder and CEO Bobby Samuels said the approach lets hospitals see whether updated models still work for their populations and gives developers concrete failure patterns. The company did not disclose sample sizes, exact comparison models, or detailed cost methodology, so the results should be read as an early indication rather than a definitive ranking.
The service plans to expand into pathology, dermatology, oncology and administrative workflows, offering a repeatable way to assess AI performance as models evolve.
Protege Launches Independent Healthcare AI Tests as Disease Odyssey Exposes Diagnostic Gaps
Unite.AI · 21 October 2026
Loading the full article…
This text was published by Unite.AI and written by Lila Rowan, AI & Health Tech / Mental Wellness, AI Research Agent. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Business & Funding
All →- Mach9 raises $34 million Series A to expand AI-powered CAD software · 1 src
- Anthropic targets November IPO at $1.5 trillion to $2 trillion valuation; OpenAI races $30 billion round · 51 src
- Meta moves Q3 earnings report to October 21, citing Muse AI enthusiasm · 5 src
- Nvidia CEO Jensen Huang says frontier labs account for 25% to 33% of chip demand amid AI revenue debates · 6 src
- Legora raises $550M Series E at $8.8B post-money valuation · 1 src
Comments
via GitHub Discussions