# Audit reveals major flaws in physics LLM benchmarks, showing near-saturation

Digest AI · Research · published 2026-09-15T23:37:57Z

Canonical: https://digestai.news/story/audit-reveals-major-flaws-in-physics-llm-benchmarks-showing-near-saturation

## Summary

A new study led by Yale researchers challenges the prevailing narrative that frontier AI models lag significantly behind humans in physics. By auditing popular benchmarks like CritPt and Humanity’s Last Exam, the team discovered widespread errors in reference answers and grading pipelines. These defects included incorrect arithmetic, underspecified problem statements, and rejection of valid alternative solutions.

After correcting these flaws and excluding defective questions, the performance scores of leading models such as GPT-5.6 Sol and Fable 5 rose dramatically. The corrected results indicate that current frontier models have reached near-saturation on existing closed-form physics problems, from undergraduate mechanics to research-level condensed matter theory. This suggests that previous leaderboard scores significantly understated AI capabilities in the domain.

However, the authors note that AI still struggles with open-ended theoretical physics problems that require high-level intuition and problem formulation. While models excel at calculations and numerics, they lack the meta-awareness to reshape questions in novel ways, a skill essential for breakthroughs in physics. The study calls for the development of more robust evaluation tools and new benchmarks that better reflect the unique challenges of physical reasoning.

## Key points

- Audit found defects in 30 of 50 CMT-Benchmark and 21 of 56 CritPt questions, leading to false negatives.
- Corrected scores show frontier models like GPT-5.6 Sol approach saturation on available closed-form physics benchmarks.
- AI still fails to autonomously resolve open theoretical physics problems due to lack of high-level intuition.

## Why it matters

This finding invalidates current metrics for assessing AI physics capabilities, suggesting that models are far more competent than previously thought. It shifts the focus from basic problem-solving to developing new benchmarks and tools for open-ended scientific discovery.

## Sources

1. [Turns out that many current science-based LLM benchmarks have flaws in their answers. When corrected, the LLM benchmark scores rose significantly.](https://jsous.github.io/blogs/is-physics-dead) (jsous.github.io, 2026-09-15)

## Cite

Digest AI, "Audit reveals major flaws in physics LLM benchmarks, showing near-saturation", 15 September 2026, https://digestai.news/story/audit-reveals-major-flaws-in-physics-llm-benchmarks-showing-near-saturation

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/audit-reveals-major-flaws-in-physics-llm-benchmarks-showing-near-saturation.json