Inception v3 CNN beats GPT-6 Astra and Claude Fable 5.1 on skin cancer test

The study highlights that specialized, smaller models can outperform general-purpose frontier LLMs in specific medical imaging tasks. The CNN was deliberately not state-of-the-art, using an outdated architecture, yet it outperformed the newer models on most categories, including melanoma and benign moles. The LLMs showed distinct biases, with GPT-6 Astra over-diagnosing basal cell carcinoma and Claude Fable 5.1 over-diagnosing actinic keratosis. When asked as a patient whether to see a doctor, both LLMs advised a visit in almost every case, regardless of the lesion's benign nature.

The author notes several caveats, including the small sample size of 200 lesions and the possibility that the LLMs may have seen the public dataset during training. The CNN is small enough to run in a phone browser using WebGPU, and all code, prompts, and results are available for public review. The analysis was assisted by Claude Code, one of the models tested.