Researchers test six AI tools on a 102-article scientific literature review

The tools struggled with accuracy in classifying articles. Three tools correctly identified all 18 articles discussing the Spanish plume substantially but misclassified 40% to 79% of the 84 passing mentions. Their synthesized summaries, reviewed blindly by domain experts, were largely accurate but lacked depth, contained unsupported claims, quotation errors, and exaggerated consensus statements. All tools proposed overly broad research directions, though two aligned with questions from the benchmark review. The study concludes that AI can assist with discovery and interpretation but cannot replace expert judgment in synthesizing evidence.