{"version":1,"type":"story","url":"https://digestai.news/story/arxiv-paper-finds-shortcuts-distort-rag-benchmarking-accuracy","json":"https://digestai.news/story/arxiv-paper-finds-shortcuts-distort-rag-benchmarking-accuracy.json","markdown":"https://digestai.news/story/arxiv-paper-finds-shortcuts-distort-rag-benchmarking-accuracy.md","slug":"arxiv-paper-finds-shortcuts-distort-rag-benchmarking-accuracy","headline":"arXiv paper finds shortcuts distort RAG benchmarking accuracy","summary":"A new arXiv paper critiques how conversational RAG benchmarks like MTRAG and RETECO evaluate retrieval systems. The study shows that a common shortcut—treating contexts without gold documents as unanswerable—distorts precision scores by up to 60% in some cases. Researchers re-ran official baselines on RETECO track 2 using four retrievers and three Qwen3 readers, finding that stronger retrievers often miss answers hidden in non-gold contexts, such as hard negatives or adjacent chunks of gold data.\n\nThe paper recommends scoring refusals based on content labels rather than gold document presence. It also releases code, runs, and labels to support further validation. The findings challenge current evaluation practices, which may overlook useful but non-gold information.","keyPoints":["Paper critiques shortcuts in RAG benchmarks like MTRAG and RETECO, calling them inaccurate","Tests four retrievers and three Qwen3 readers on RETECO track 2 data","Recommends scoring refusals by content, not gold document presence"],"whyItMatters":"This paper exposes flaws in how RAG systems are benchmarked, risking misguided model development and wasted resources.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":["Qwen3","RETECO","MTRAG"],"people":[]},"firstPublishedAt":"2026-11-02T05:00:00Z","updatedAt":"2026-11-02T05:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.CL","title":"Better Retrievers, Worse Labels: How Incomplete Relevance Judgments Distort Abstention Evaluation in Conversational RAG","url":"https://arxiv.org/abs/2610.40966","publishedAt":"2026-11-02T05:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"arXiv paper finds shortcuts distort RAG benchmarking accuracy\", 2 November 2026, https://digestai.news/story/arxiv-paper-finds-shortcuts-distort-rag-benchmarking-accuracy","publisher":"Digest AI","title":"arXiv paper finds shortcuts distort RAG benchmarking accuracy","datePublished":"2026-11-02T05:00:00Z","url":"https://digestai.news/story/arxiv-paper-finds-shortcuts-distort-rag-benchmarking-accuracy"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}