Apple introduces CapQuiz to evaluate video caption quality via multiple-choice questions
Apple Machine Learning Research has introduced CapQuiz, a new reference-free benchmark designed to assess the quality of video captions generated by Visual Large Language Models (VLLMs). Traditional evaluation methods often penalize valid captions due to lexical mismatches with ground-truth references, failing to capture the subjective nature of video description. CapQuiz addresses this by…
Key points
- Apple introduces CapQuiz, a reference-free benchmark evaluating video captions via multiple-choice questions.
- The benchmark covers 10 question types across 24 video domains to measure factuality and coverage.
- CapQuiz correlates better with human judgments than traditional text-matching metrics for VLLMs.
The benchmark utilizes a hierarchical taxonomy of 10 question types, spanning descriptive and inferential categories, across 24 diverse video domains. Instead of relying on text matching, CapQuiz evaluates captions based on their utility in answering human-verified, fine-grained multiple-choice questions derived directly from the video content. This approach provides a more granular and interpretable analysis of model performance compared to one-dimensional metrics.
The researchers also proposed CapF1, a composite metric that synthesizes CapP (factuality) and CapR (coverage). Extensive experiments indicate that CapQuiz correlates significantly better with human judgments than existing standards. This work aims to provide a more robust framework for evaluating multimodal models, ensuring that generated descriptions are both comprehensive and accurate.
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
Apple Machine Learning Research · 11 September 2026
Putting Captions to the Test: Evaluating Video Caption Quality through Multiple-Choice Question Answering
AuthorsZizhen Wang, Bo Feng, Zhengfeng Lai†**, Shiyu Li, Yang Lu, Meng Cao, Ping Huang, Simon Wang
Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated text against ground-truth references. This paradigm suffers from the “one-to-many” nature of video description, where high-quality captions are often penalized for lexical mismatches or valid shifts in visual focus. Furthermore, such assessments are typically one-dimensional, failing to provide a fine-grained analysis of caption quality. To address this, we redefine caption quality via information fidelity: A caption must maximize the coverage of salient visual information while ensuring strict factuality. We introduce CapQuiz, a novel reference-free benchmark that assesses captions based on their utility in answering human-verified, fine-grained, multiple-choice questions derived from the video. CapQuiz features a hierarchical taxonomy of 10 question types (spanning Descriptive and Inferential categories) across 24 diverse video domains. We further formulate CapF1, a composite metric that synthesizes CapP (measuring factuality) and CapR (measuring coverage). Extensive experiments demonstrate that CapQuiz correlates significantly better with human judgments than existing metrics and offers interpretable insights into model performance.
BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning
May 11, 2026research area Computer Vision, research area Methods and Algorithms
Image captioning is one of the most fundamental tasks in computer vision. Owing to its open-ended nature, it has received significant attention in the era of multimodal large language models (MLLMs). In pursuit of ever more detailed and accurate captions, recent work has increasingly turned to reinforcement learning (RL). However, existing captioning-RL methods and evaluation metrics often emphasize a narrow notion of caption quality, inducing…
Revisit Large-Scale Image–Caption Data in Pre-training Multimodal Foundation Models
April 8, 2025research area Computer Vision, research area Methods and Algorithmsconference ICLR
Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. Notably, the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still unclear. Additionally, different multimodal foundation models may have distinct preferences for specific caption formats while the efforts of studying the optimal captions for each foundation…
This text was published by Apple Machine Learning Research . It is reproduced here with attribution so you can read it in full; the rights remain with the publisher. Read it at the source ↗
The headline, key points and digest above were generated by Digest AI's editorial model from the linked sources. Automated summaries can contain errors: the sources are the record. Spotted a mistake? Tell us.
Comments
via GitHub Discussions