Researchers introduce NewsBench to evaluate AI news quality
Researchers have released NewsBench, a benchmark that measures the epistemic quality of AI‑generated news. The tool focuses on three editorial dimensions—source quality, factuality, and neutrality—using a two‑phase pipeline where domain experts set guidelines and AI judges are calibrated against expert‑annotated gold labels.
Key points
- NewsBench benchmark uses ~3,000 prompts and 2,500 expert annotations
- It evaluates source quality, factuality, neutrality via expert‑guided AI judges
- Applied to four frontier models, revealing distinct failure profiles
NewsBench contains roughly 3,000 evaluation prompts, about 500 of which are refreshed each month to cover emerging events. The benchmark also includes 2,500 human expert‑annotated items for calibration and validation, and a suite of AI judges that enable scalable assessment.
The authors applied NewsBench to four frontier AI models, uncovering distinct and non‑overlapping failure profiles across sourcing, factuality, and neutrality, and reporting inter‑rater reliability and AI‑human agreement statistics.
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Researchers propose synthetic document finetuning to detect alignment-faking LLMs · 3 src
- Researchers introduce LOTUS method to detect LLM hallucinations via low-rank decomposition · 1 src
- Researchers find Instruction tuning sharpens Number-Choice heuristics in language models · 1 src
- Researchers release FinLLM-KG framework for financial sentiment analysis using knowledge graphs · 1 src
- Researchers release MedJev, a zero-shot medical AI model for laptops · 1 src
Comments
via GitHub Discussions