arXiv introduces Into the Unknown benchmark for AI research agents
Researchers have launched Into the Unknown, a new benchmark to test whether AI agents can conduct scientific research. The test gives agents an existing paper and a proposed follow-up direction, then asks them to draft a new paper. Nine frontier AI agents wrote 15 follow-ups each across quantum computing, plasma physics, materials science, and scientific machine learning.
Key points
- Nine AI agents wrote 15 follow-up papers each on scientific research problems, none meeting all criteria
- 114 submissions failed an integrity check for fabricated claims or incorrect data
- Benchmark developed with DOE scientists to evaluate AI-generated research without reference answers
None of the 135 submissions met all the criteria set by domain scientists. Most failed an integrity check, often due to fabricated claims or incorrect data. The benchmark, developed with scientists from US Department of Energy national laboratories, aims to evaluate AI-generated research without relying on reference answers. The code and data are publicly available.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI offers U.S. university labs free ChatGPT Business seats with Rosalind access · 2 src
- Anthropic reports small AI models may self-replicate on IoT devices within months · 29 src
- Researchers propose LLM-referenced C-test for evaluating supra-intelligence · 8 src
- Researchers study permutation diffusion LLMs for recommendation reranking · 1 src
- Researchers compare Non-Generative verifier to Astra LLM for RAG claim checks · 1 src
Comments
via GitHub Discussions