# Researchers test six AI tools on a 102-article scientific literature review

Digest AI · Research · published 2026-11-05T05:00:00Z

Canonical: https://digestai.news/story/researchers-test-six-ai-tools-on-a-102-article-scientific-literature-r

## Summary

A new study on arXiv evaluates how well six AI tools handle the full process of reviewing scientific literature, not just individual tasks. The benchmark uses a published review of the Spanish plume weather pattern and its 102-article corpus. In the initial search phase, five tools collectively found 57 articles, with the best tool identifying 42. With institutional library access, two tools—Codex and Claude Code—reached 82 and 79 articles respectively, though neither matched the full corpus of 102.

The tools struggled with accuracy in classifying articles. Three tools correctly identified all 18 articles discussing the Spanish plume substantially but misclassified 40% to 79% of the 84 passing mentions. Their synthesized summaries, reviewed blindly by domain experts, were largely accurate but lacked depth, contained unsupported claims, quotation errors, and exaggerated consensus statements. All tools proposed overly broad research directions, though two aligned with questions from the benchmark review. The study concludes that AI can assist with discovery and interpretation but cannot replace expert judgment in synthesizing evidence.

## Key points

- Six AI tools tested on a 102-article scientific literature review benchmark, covering discovery, interpretation, synthesis, and research proposals
- Best tool found 42 articles without institutional access; with access, Codex and Claude Code reached 82 and 79 articles respectively
- Three tools misclassified 40%–79% of passing mentions and produced syntheses with unsupported claims and quotation errors

## Why it matters

This study highlights both the potential and limitations of AI in scientific literature reviews, showing progress in discovery but persistent risks of misclassification and overgeneralization in synthesis. Researchers may need to adjust workflows to balance AI assistance with human oversight.

## Sources

1. [Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark](https://arxiv.org/abs/2611.05459) (arXiv cs.AI, 2026-11-05, primary source)

## Cite

Digest AI, "Researchers test six AI tools on a 102-article scientific literature review", 5 November 2026, https://digestai.news/story/researchers-test-six-ai-tools-on-a-102-article-scientific-literature-r

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-test-six-ai-tools-on-a-102-article-scientific-literature-r.json