Researchers introduce Unsatz benchmark for LLM scientific discovery

The benchmark tests agents across 80 systems in 20 domains. Agents must buy experiments within a budget, write candidate programs in a specific modeling language, and are scored on their ability to predict unseen experiments, such as interventions or conditions outside the explored range. Results show that nine different language models solved between 21% and 60% of the tasks. Notably, classical equation-discovery methods almost never succeeded on 44 of these systems, even when the true equations were provided with unknown coefficients, highlighting that the primary difficulty lies in finding the correct structure rather than estimating parameters.

The study also found that providing agents with the program discovered for a related system only helped when the new system shared the same mechanism and the initial investigation was successful. Furthermore, while stronger agents made fewer unscientific errors, they tended to prefer revisions that lowered training error over those that improved predictive performance, indicating a persistent challenge in balancing fit with generalization.