# Researchers release ListQA benchmark for testing LLMs on multi-fact list retrieval

Digest AI · Research · published 2026-10-15T04:00:00Z

Canonical: https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li

## Summary

Researchers published ListQA, a new benchmark for evaluating how well large language models handle questions requiring multiple facts in structured lists. The dataset includes 9,045 human-curated questions across eight categories and over 60 countries, with an error rate of 1.7%. Examples range from flat lists like ‘Which NCAA teams went undefeated between 2000 and 2024?’ to hierarchical questions with sub-attributes, such as ‘...and what was their record and result?’ The questions are designed to remain valid over time, avoiding periodic updates.

The benchmark tests models on factual accuracy, hallucination, and format compliance. Researchers evaluated 34 models (8 families, 42 configurations) and found that even the best-performing models scored below 37% on LLM-Judge-F1. Providing the source Wikipedia page significantly improved scores, suggesting models struggle to retrieve facts independently. The dataset and evaluation code are publicly released for further study.

## Key points

- ListQA benchmark includes 9,045 human-curated questions requiring multi-fact list retrieval across 60+ countries
- Best-performing models score below 37% on LLM-Judge-F1, with hierarchical questions proving harder overall
- Source Wikipedia pages improve model performance, indicating retrieval challenges from parameters alone

## Why it matters

ListQA fills a gap in evaluating LLMs’ ability to organize and retrieve structured factual knowledge, critical for real-world applications like education, research, and customer support where precise multi-fact responses matter.

## Sources

1. [ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models](https://arxiv.org/abs/2610.19165) (arXiv cs.AI, 2026-10-15, primary source)

## Cite

Digest AI, "Researchers release ListQA benchmark for testing LLMs on multi-fact list retrieval", 15 October 2026, https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/researchers-release-listqa-benchmark-for-testing-llms-on-multi-fact-li.json