Audit reveals major flaws in physics LLM benchmarks, showing near-saturation
A new study led by Yale researchers challenges the prevailing narrative that frontier AI models lag significantly behind humans in physics. By auditing popular benchmarks like CritPt and Humanity’s Last Exam, the team discovered widespread errors in reference answers and grading pipelines. These defects included incorrect arithmetic, underspecified problem statements, and rejection of valid…
Key points
- Audit found defects in 30 of 50 CMT-Benchmark and 21 of 56 CritPt questions, leading to false negatives.
- Corrected scores show frontier models like GPT-5.6 Sol approach saturation on available closed-form physics benchmarks.
- AI still fails to autonomously resolve open theoretical physics problems due to lack of high-level intuition.
After correcting these flaws and excluding defective questions, the performance scores of leading models such as GPT-5.6 Sol and Fable 5 rose dramatically. The corrected results indicate that current frontier models have reached near-saturation on existing closed-form physics problems, from undergraduate mechanics to research-level condensed matter theory. This suggests that previous leaderboard scores significantly understated AI capabilities in the domain.
However, the authors note that AI still struggles with open-ended theoretical physics problems that require high-level intuition and problem formulation. While models excel at calculations and numerics, they lack the meta-awareness to reshape questions in novel ways, a skill essential for breakthroughs in physics. The study calls for the development of more robust evaluation tools and new benchmarks that better reflect the unique challenges of physical reasoning.
Model page: GPT-6 Astra →
Turns out that many current science-based LLM benchmarks have flaws in their answers. When corrected, the LLM benchmark scores rose significantly.
jsous.github.io · 15 September 2026Loading the full article…
This text was published by jsous.github.io. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
Coverage and discussion
1source- Reddit discussionreddit.com
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- MIT researchers show neural networks can build own learning rules · 1 src
- Anthropic research suggests small AI models may self-replicate on IoT devices · 6 src
- Anthropic co-founder predicts full RSI possible by 2027 · 1 src
- Sakana AI’s MASS method improves model self-judgment but risks baking errors into training · 1 src
- Microsoft Research finds clearing chat history boosts AI agent performance by 66% · 1 src
Comments
via GitHub Discussions