Apple introduces CapQuiz to evaluate video caption quality via multiple-choice questions
Apple Machine Learning Research has introduced CapQuiz, a new reference-free benchmark designed to assess the quality of video captions generated by Visual Large Language Models (VLLMs). Traditional evaluation methods often penalize valid captions due to lexical mismatches with ground-truth references, failing to capture the subjective nature of video description. CapQuiz addresses this by…
Key points
- Apple introduces CapQuiz, a reference-free benchmark evaluating video captions via multiple-choice questions.
- The benchmark covers 10 question types across 24 video domains to measure factuality and coverage.
- CapQuiz correlates better with human judgments than traditional text-matching metrics for VLLMs.
The benchmark utilizes a hierarchical taxonomy of 10 question types, spanning descriptive and inferential categories, across 24 diverse video domains. Instead of relying on text matching, CapQuiz evaluates captions based on their utility in answering human-verified, fine-grained multiple-choice questions derived directly from the video content. This approach provides a more granular and interpretable analysis of model performance compared to one-dimensional metrics.
The researchers also proposed CapF1, a composite metric that synthesizes CapP (factuality) and CapR (coverage). Extensive experiments indicate that CapQuiz correlates significantly better with human judgments than existing standards. This work aims to provide a more robust framework for evaluating multimodal models, ensuring that generated descriptions are both comprehensive and accurate.
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Apple Machine Learning Research · 11 September 2026
Loading the full article…
This text was published by Apple Machine Learning Research. It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us. Published by Martin K., who runs Digest AI and handles corrections.
More in Research
All →- Anthropic research suggests small AI models may self-replicate on IoT devices · 7 src
- Researcher predicts AI engineering acceleration but not superintelligence · 1 src
- MIT researchers show neural networks can build own learning rules · 1 src
- Anthropic co-founder predicts full RSI possible by 2027 · 1 src
- Sakana AI’s MASS method improves model self-judgment but risks baking errors into training · 1 src
Comments
via GitHub Discussions