{"version":1,"type":"story","url":"https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility","json":"https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility.json","markdown":"https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility.md","slug":"openai-releases-paperbench-benchmark-for-ai-research-reproducibility","headline":"OpenAI releases PaperBench benchmark for AI research reproducibility","summary":"OpenAI released PaperBench, a benchmark to measure AI reproducibility of research papers. It covers 20 papers from ICML 2024, spanning 12 fields. The benchmark breaks each paper into 8,316 scorable items, allowing partial scoring.\n\nThe evaluation has three stages: code creation, execution in a new environment, and result scoring. The highest average reproduction score achieved was 21.0% by an agent combining Claude 3.5 Sonnet with open-source tools. Human experts outperformed AI, and the benchmark shows AI still lags behind human baseline.\n\nAutomated grading was also tested. JudgeEval achieved an F1 of 0.83. The study highlights gaps in AI's ability to fully reproduce research and suggests a collaborative workflow where AI handles preparation and humans verify results.","keyPoints":["OpenAI's PaperBench benchmark covers 20 ICML 2024 papers and 8,316 scoring items.","Highest average reproduction score 21.0% using Claude 3.5 Sonnet with open-source tools.","Human experts outperformed AI; AI did not exceed human baseline."],"whyItMatters":"PaperBench provides a systematic way to gauge AI agents' ability to reproduce scientific work, revealing a significant gap between current models and human researchers. The benchmark informs future AI development, encourages better reproducibility practices, and helps the community set realistic expectations for AI-driven research automation.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":["OpenAI"],"models":["Claude 3.5 Sonnet"],"people":[]},"firstPublishedAt":"2026-11-03T00:04:00Z","updatedAt":"2026-11-03T00:04:00Z","sourceCount":1,"hasPrimarySource":false,"sources":[{"outlet":"note.com","title":"Can AI Reproduce Research Papers? The Reality Measured by OpenAI's \"PaperBench\"","url":"https://note.com/turuto/n/nbbca9c9fd41c?hl=en","publishedAt":"2026-11-03T00:04:00Z","type":"press","primary":false,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"OpenAI releases PaperBench benchmark for AI research reproducibility\", 3 November 2026, https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility","publisher":"Digest AI","title":"OpenAI releases PaperBench benchmark for AI research reproducibility","datePublished":"2026-11-03T00:04:00Z","url":"https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}