Study shows Conformal factuality control improves claim support in Multi-Hop RAG
Researchers tested claim-level conformal factuality control in multi-hop retrieval-augmented generation using Llama 3.1 8B and GPT-4o-mini on HotpotQA, Natural Questions, and TriviaQA. At a 95% conformal target, the fraction of responses with fully supported claims rose to between 95.80% and 97.20%, up from 55.60%-76.03% without filtering. However, this improvement came with low claim retention:…
Key points
- Conformal factuality control increased fully supported claims to 95.80%-97.20% at 95% target
- Without filtering, supported claims ranged from 55.60% to 76.03%
- Claim retention was low: only 4.41%-31.09% of claims retained at 95% target
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic's Claude begins 'dreaming' to reorganize memory in tests · 1 src
- John Langford quits Microsoft, declares AI Frontiers over · 1 src
- Anthropic releases study on Claude usage in scientific research · 3 src
- MIT and BASF release CheMeleon for low-data molecular prediction · 2 src
- Anthropic tests Claude agents in EVE Online trading and sabotage detection · 1 src
Comments
via GitHub Discussions