RefCaptioner: Multi-Reference Image-Grounded Video Captioning

2026-07-30

AI Summary

This work addresses the challenge that existing video captioning models struggle to explicitly align local visual content with multi-reference images at the phrase level during generation. We introduce a novel task of multi-reference image–guided video captioning and contribute a large-scale text-image-video corpus along with the MRVBench evaluation benchmark. To tackle this task, we propose RefCaptioner, a two-stage post-training framework that integrates mixed-data supervised fine-tuning and hierarchical coverage-discounted GRPO reinforcement learning to jointly optimize reference selection, phrase grounding, distractor suppression, and cross-reference consistency. Experiments demonstrate that our approach achieves state-of-the-art overall performance among open-source models, remains competitive on standard benchmarks, and generates captions consistently preferred in human evaluations, while also enabling high-fidelity video reconstruction.

Abstract

Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captioning, a new task requiring factual video descriptions with phrase-level reference grounding, and propose RefCaptioner, a two-stage post-training framework for this task. RefCaptioner combines mixed-data SFT with Hierarchical Coverage-Discounted GRPO to jointly improve reference selection, phrase-level binding, distractor rejection, and cross-reference consistency while preserving general video-captioning ability. To support training, we construct a corpus containing $20,000$ videos and 171,354 reference images. We further introduce MRVBench, a benchmark for evaluating caption factuality and multi-reference grounding on both real-world and AI-generated videos. Experiments show that RefCaptioner achieves the best overall performance among the open-source models while remaining competitive on standard video captioning benchmarks. Human evaluation further confirms that its captions are preferred by annotators and enable more source-faithful video reconstruction with both open-source and proprietary video generators.

Problem

• video captioning
• reference grounding
• multi-reference
• phrase-level grounding
• factual description

Innovation

• multi-reference grounding
• video captioning
• phrase-level binding
• hierarchical coverage-discounted GRPO
• factuality evaluation