Researchers release ListQA benchmark for testing LLMs on multi-fact list retrieval
Researchers published ListQA, a new benchmark for evaluating how well large language models handle questions requiring multiple facts in structured lists. The dataset includes 9,045 human-curated questions across eight categories and over 60 countries, with an error rate of 1.7%. Examples range from flat lists like ‘Which NCAA teams went undefeated between 2000 and 2024?’ to hierarchical…
Key points
- ListQA benchmark includes 9,045 human-curated questions requiring multi-fact list retrieval across 60+ countries
- Best-performing models score below 37% on LLM-Judge-F1, with hierarchical questions proving harder overall
- Source Wikipedia pages improve model performance, indicating retrieval challenges from parameters alone
The benchmark tests models on factual accuracy, hallucination, and format compliance. Researchers evaluated 34 models (8 families, 42 configurations) and found that even the best-performing models scored below 37% on LLM-Judge-F1. Providing the source Wikipedia page significantly improved scores, suggesting models struggle to retrieve facts independently. The dataset and evaluation code are publicly released for further study.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- OpenAI offers U.S. university labs free ChatGPT Business seats with Rosalind access · 2 src
- Anthropic reports small AI models may self-replicate on IoT devices within months · 29 src
- Researchers propose LLM-referenced C-test for evaluating supra-intelligence · 8 src
- Researchers study permutation diffusion LLMs for recommendation reranking · 1 src
- Researchers compare Non-Generative verifier to Astra LLM for RAG claim checks · 1 src
Comments
via GitHub Discussions