Gemma-3-12b fails to detect misinformation when poisoned claims dominate retrieved evidence
Researchers found that the language model Gemma-3-12b struggles to identify misinformation when most of its retrieved documents support a false claim. The study, published on arXiv, tested 200 questions from the SciFact dataset, comparing scenarios where one document was poisoned against six healthy ones, versus six poisoned documents with one healthy one.
Key points
- Gemma-3-12b’s misinformation detection drops from 25% to 10% when poisoned claims dominate retrieved evidence
- 34 of 38 questions shifted from detected to undetected in majority-poisoned scenarios, per the study’s analysis
- Researchers attribute the trend to a ‘conformity bias’ in RAG-based models, per arXiv paper
The model’s ability to detect misinformation dropped sharply: it correctly flagged only 25% of poisoned claims in the minority condition (50/200) but just 10% (20/200) when poisoned claims dominated. Of the 38 questions with differing outcomes, 34 shifted from detected to undetected, while only four reversed. The researchers call this a ‘conformity bias’—the model conforms to the majority of retrieved evidence, even when it contradicts facts. The study highlights a critical flaw in retrieval-augmented generation (RAG), where models may prioritize consensus over accuracy.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- SemiAnalysis releases GDPx benchmark with Claude Opus 5.5 leading at 12% · 3 src
- Researchers release Qistas protocol for testing LLM fairness across cultures · 1 src
- Researchers compare speculative drafters and continuous diffusion language models for text generation · 1 src
- Researchers propose Li²SAO to improve asynchronous RL training for LLMs · 1 src
- Researchers study how LLMs assign emotional value to outcomes · 1 src
Comments
via GitHub Discussions