{"version":1,"type":"story","url":"https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li","json":"https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li.json","markdown":"https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li.md","slug":"researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li","headline":"Researchers release ListQA benchmark for testing LLMs on multi-fact list retrieval","summary":"Researchers published ListQA, a new benchmark for evaluating how well large language models handle questions requiring multiple facts in structured lists. The dataset includes 9,045 human-curated questions across eight categories and over 60 countries, with an error rate of 1.7%. Examples range from flat lists like ‘Which NCAA teams went undefeated between 2000 and 2024?’ to hierarchical questions with sub-attributes, such as ‘...and what was their record and result?’ The questions are designed to remain valid over time, avoiding periodic updates.\n\nThe benchmark tests models on factual accuracy, hallucination, and format compliance. Researchers evaluated 34 models (8 families, 42 configurations) and found that even the best-performing models scored below 37% on LLM-Judge-F1. Providing the source Wikipedia page significantly improved scores, suggesting models struggle to retrieve facts independently. The dataset and evaluation code are publicly released for further study.","keyPoints":["ListQA benchmark includes 9,045 human-curated questions requiring multi-fact list retrieval across 60+ countries","Best-performing models score below 37% on LLM-Judge-F1, with hierarchical questions proving harder overall","Source Wikipedia pages improve model performance, indicating retrieval challenges from parameters alone"],"whyItMatters":"ListQA fills a gap in evaluating LLMs’ ability to organize and retrieve structured factual knowledge, critical for real-world applications like education, research, and customer support where precise multi-fact responses matter.","category":{"slug":"research","name":"Research","url":"https://digestai.news/category/research"},"entities":{"companies":[],"models":[],"people":[]},"firstPublishedAt":"2026-10-15T04:00:00Z","updatedAt":"2026-10-15T04:00:00Z","sourceCount":1,"hasPrimarySource":true,"sources":[{"outlet":"arXiv cs.AI","title":"ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models","url":"https://arxiv.org/abs/2610.19165","publishedAt":"2026-10-15T04:00:00Z","type":"primary","primary":true,"lead":true}],"sourceNotes":null,"discussions":[],"thread":null,"cite":{"text":"Digest AI, \"Researchers release ListQA benchmark for testing LLMs on multi-fact list retrieval\", 15 October 2026, https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li","publisher":"Digest AI","title":"Researchers release ListQA benchmark for testing LLMs on multi-fact list retrieval","datePublished":"2026-10-15T04:00:00Z","url":"https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li"},"generatedBy":"Written by Digest AI's editorial model from the linked sources; the sources are the record.","license":"Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse"}