ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning

2026-07-23

AI Summary

This work addresses the challenge that existing video captioning methods struggle to simultaneously ensure content completeness and factual accuracy without model retraining, often omitting critical information or generating hallucinations due to inadequate consideration of object importance. To overcome this limitation, the authors propose a training-free, iterative post-processing framework that introduces, for the first time, an object saliency-guided mechanism. This mechanism scores detected objects by integrating spatial saliency, temporal persistence, and relational dynamics, and then employs multi-round prompt-driven iterative refinement to inject high-relevance yet missing key objects into the generated captions. The approach is lightweight, scalable, and model-agnostic, requiring no parameter updates. Evaluations on MSVD and MSR-VTT demonstrate a 48% improvement in caption completeness and a 45% reduction in hallucinations under human evaluation, significantly outperforming strong baselines without relying on reference captions or model retraining.

Abstract

Improving video captioning quality typically demands retraining large vision-language models, an expensive and often impractical requirement. Existing training-free alternatives instead ground captions in detected objects to curb hallucination, but apply only a single, fixed correction pass without prioritizing which objects matter most, leaving semantically significant content omitted. We propose a prominence-aware, iterative post-hoc rectification framework that overcomes both limitations without modifying the underlying captioning model's parameters: a lightweight scoring mechanism ranks detected objects by spatial saliency, temporal persistence, and relational dynamics, and an iterative, prompt-driven refinement loop uses this ranking to progressively inject missing yet contextually relevant objects into the caption over multiple rounds. We validate the framework on MSVD and MSR-VTT using object-grounded automatic metrics, a 110-participant human study, and qualitative comparison against ChatGPT and Gemini; in human evaluation, the framework raises perceived completeness by up to 48% and reduces hallucination by up to 45% relative to a strong pretrained captioning baseline, all without retraining or reference captions. These results position prominence-guided iterative rectification as a lightweight, scalable, and model-agnostic route to more complete and trustworthy video captioning, with direct relevance to accessibility, retrieval, and other multimedia understanding applications.

Problem

• video captioning
• hallucination
• object grounding
• caption completeness
• training-free rectification

Innovation

• prominence-aware
• iterative rectification
• object grounding
• video captioning
• hallucination reduction