# OpenAI releases PaperBench benchmark for AI research reproducibility

Digest AI · Research · published 2026-11-03T00:04:00Z

Canonical: https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility

## Summary

OpenAI released PaperBench, a benchmark to measure AI reproducibility of research papers. It covers 20 papers from ICML 2024, spanning 12 fields. The benchmark breaks each paper into 8,316 scorable items, allowing partial scoring.

The evaluation has three stages: code creation, execution in a new environment, and result scoring. The highest average reproduction score achieved was 21.0% by an agent combining Claude 3.5 Sonnet with open-source tools. Human experts outperformed AI, and the benchmark shows AI still lags behind human baseline.

Automated grading was also tested. JudgeEval achieved an F1 of 0.83. The study highlights gaps in AI's ability to fully reproduce research and suggests a collaborative workflow where AI handles preparation and humans verify results.

## Key points

- OpenAI's PaperBench benchmark covers 20 ICML 2024 papers and 8,316 scoring items.
- Highest average reproduction score 21.0% using Claude 3.5 Sonnet with open-source tools.
- Human experts outperformed AI; AI did not exceed human baseline.

## Why it matters

PaperBench provides a systematic way to gauge AI agents' ability to reproduce scientific work, revealing a significant gap between current models and human researchers. The benchmark informs future AI development, encourages better reproducibility practices, and helps the community set realistic expectations for AI-driven research automation.

## Sources

1. [Can AI Reproduce Research Papers? The Reality Measured by OpenAI's "PaperBench"](https://note.com/turuto/n/nbbca9c9fd41c?hl=en) (note.com, 2026-11-03)

## Cite

Digest AI, "OpenAI releases PaperBench benchmark for AI research reproducibility", 3 November 2026, https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/openai-releases-paperbench-benchmark-for-ai-research-reproducibility.json