arXiv study tests how AI models improve data science projects with feedback
Researchers on arXiv examined whether agentic AI models can complete full data science projects and how feedback affects their performance. A ‘writer’ model generated projects—including data download, cleaning, analysis, and a short paper—while a ‘grader’ model evaluated the work. Each model served as both writer and grader, creating 49 pairs tested over three rounds of iteration.
Key points
- 7 models from 5 labs—Anthropic, OpenAI, Google, DeepSeek, Z.AI—tested as writers and graders in 49 pairs
- 107 of 147 initial submissions ran successfully, scoring 59% on average before feedback
- Scores rose to 76% after three rounds, with Haiku 4.5 improving by 46 points
Of the 147 initial submissions, 107 ran successfully. Scores rose from an average of 59% to 76% after the final revision. Improvements varied: Anthropic’s Opus 5.5 saw a 3-point gain, while Haiku 4.5 (also from Anthropic) improved by 46 points. Grading by the same model as the writer boosted scores by about 9 percentage points. The study found feedback generally helped, but weaker models did not consistently improve stronger ones.
Model pages: Opus 5.5 → · Gemini 3.8 Flash →
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic uses Claude to map bacteriophage vault protein at 3.2 Å resolution · 3 src
- RAGCompass routes retrievers by reranking passages, improves answer F1 by 28% · 1 src
- MIT researchers develop uq-daam to correct robot memory · 1 src
- Qwen improves LLM-based OCR correction with veto gate · 1 src
- Study: LLMs prefer to answer over clarifying ambiguous references · 1 src
Comments
via GitHub Discussions