Qwen models struggle with outdated facts in retrieved evidence
A new study on arXiv finds that AI agents often fail not because they miss relevant facts but because they cannot distinguish outdated information from correct updates. Researchers tested Qwen2.5-7B-Instruct and Qwen3-14B on the FactConsolidation task, where accuracy dropped sharply when evidence sets included competing, older statements—even if the newest fact was present.
Key points
- Qwen2.5-7B-Instruct accuracy drops from 45% to 32% when evidence budget grows from 64 to 512 tokens
- Obsolete facts hurt performance more than unrelated evidence of equal size for Qwen2.5 and Qwen3-14B
- Study uses FactConsolidation task from MemoryAgentBench to isolate retrieval vs. fact-staleness issues
With a 64-token evidence budget, Qwen2.5-7B-Instruct scored 45% accuracy, but performance plummeted to 32% when the budget expanded to 512 tokens. The same issue appeared in a 32K-history test, where BM25-64 outperformed BM25-512. The study suggests that retrieval alone cannot guarantee correct answers if agents cannot filter out stale information.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers propose synthetic document finetuning to detect alignment-faking LLMs · 3 src
- Researchers introduce LOTUS method to detect LLM hallucinations via low-rank decomposition · 1 src
- Researchers find Instruction tuning sharpens Number-Choice heuristics in language models · 1 src
- Researchers release FinLLM-KG framework for financial sentiment analysis using knowledge graphs · 1 src
- Researchers release MedJev, a zero-shot medical AI model for laptops · 1 src
Comments
via GitHub Discussions