{"version":1,"type":"story","url":"https://digestai.news/story/protege-launches-independent-healthcare-ai-benchmark-tests","json":"https://digestai.news/story/protege-launches-independent-healthcare-ai-benchmark-tests.json","markdown":"https://digestai.news/story/protege-launches-independent-healthcare-ai-benchmark-tests.md","slug":"protege-launches-independent-healthcare-ai-benchmark-tests","headline":"Protege launches independent healthcare AI benchmark tests","summary":"Protege announced a new service that runs independent, double‑blind evaluations of healthcare AI on patient records the models have never seen. Its first benchmark, called Disease Odyssey, showed that leading models identified conditions such as sleep apnea and anemia in only 53%–69% of cases when working from longitudinal histories. The launch also featured tests of Xlue AI’s HealthSAW model and Standard Model Biomedicine, with the latter reporting prediction costs 36–170 times lower than competing systems and HealthSAW costing less than 0.1% of the leading models per patient.\n\nProtege’s platform uses privacy‑protected records, clinician‑verified answers, and versioned tasks to keep test material separate from development data, aiming to prevent benchmark over‑fitting. Co‑founder and CEO Bobby Samuels said the approach lets hospitals see whether updated models still work for their populations and gives developers concrete failure patterns. The company did not disclose sample sizes, exact comparison models, or detailed cost methodology, so the results should be read as an early indication rather than a definitive ranking.\n\nThe service plans to expand into pathology, dermatology, oncology and administrative workflows, offering a repeatable way to assess AI performance as models evolve.","keyPoints":["Protege's Disease Odyssey benchmark found leading models identified conditions in only 53%–69% of cases.","Standard Model Biomedicine and Xlue AI's HealthSAW claimed 36–170× lower prediction cost and outperformed on most tests.","Protege uses double‑blind, privacy‑protected patient records to evaluate AI, aiming to give hospitals and developers clearer performance data."],"whyItMatters":"Real‑world, independent testing reveals gaps in current healthcare AI accuracy and cost, helping hospitals assess safety and developers improve models before clinical use.","category":{"slug":"business","name":"Business & Funding","url":"https://digestai.news/category/business"},"entities":{"companies":["Protege","Xlue AI","Standard Model Biomedicine"],"models":["HealthSAW","Standard Model Biomedicine"],"people":["Bobby Samuels"]},"firstPublishedAt":"2026-10-21T13:00:09Z","updatedAt":"2026-10-21T13:00:09Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"Unite.AI","title":"Protege Launches Independent Healthcare AI Tests as Disease Odyssey Exposes Diagnostic Gaps","url":"https://unite.ai/protege-launches-independent-healthcare-ai-tests-as-disease-odyssey-exposes-diagnostic-gaps","publishedAt":"2026-10-21T13:00:09Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Protege launches independent healthcare AI benchmark tests\", 21 October 2026, https://digestai.news/story/protege-launches-independent-healthcare-ai-benchmark-tests","publisher":"Digest AI","title":"Protege launches independent healthcare AI benchmark tests","datePublished":"2026-10-21T13:00:09Z","url":"https://digestai.news/story/protege-launches-independent-healthcare-ai-benchmark-tests"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}