PercepCap: Video Captioner with Structured Spatio-Temporal Perception

2026-07-22

AI Summary

This work addresses the limitations of existing video captioning methods, which lack fine-grained spatiotemporal perception and offer little interpretability for error diagnosis. The authors propose PercepCap, a novel framework that introduces explicit, structured spatiotemporal awareness into the captioning pipeline through a “perceive-then-describe” generation paradigm: it first constructs a perceptual trace encoding object trajectories and temporal events, then generates captions grounded in this evidence. Alignment and joint optimization between perception and description are achieved via a Caption-Anchored Perception data construction strategy and a two-stage training protocol—comprising perception-to-caption supervised fine-tuning followed by perception-augmented reinforcement learning. Experiments demonstrate that PercepCap significantly outperforms the Qwen3-VL baseline on both direct captioning and caption-to-question answering tasks, establishing state-of-the-art description quality.

Abstract

Video captioning requires fine-grained spatio-temporal understanding of videos, including spatial perception of where objects are located and temporal perception of when events occur. Existing MLLMs usually generate captions directly from video inputs without exposing the perceptual evidence behind descriptions. As a result, mistakes in spatiotemporal perception are only observed in the final caption, making it difficult to identify the underlying perceptual errors directly. To address these issues, we present PercepCap, a perception-aware video captioning framework that makes perceptual evidence explicit before producing the final caption. Specifically, PercepCap follows a perceive-describe generation chain, where the model first produces a spatiotemporal perception trace comprising object trajectories and temporal events, and then generates the final caption conditioned on the perceived evidence. To support this, we design a two-stage training strategy. Perceive-then-Describe Supervised Fine-tuning adapts the model from caption-only generation to the proposed perceive-describe chain, while Perception-Grounded Reinforcement Learning optimizes perception trace and caption quality with joint rewards over perception chain and the final caption. To support our two-stage training, we introduce Caption-Anchored Perception Data Construction. This pipeline builds the SFT and RL training data by first generating a caption-only description, extracting the objects and events it mentions, and grounding them back in the video with boxes and timestamps. This yields caption-aligned perception data that provides solid training ground truth, ensuring that the explicit perception trace and final caption refer to the same objects and events. Across direct caption and caption-to-QA evaluation, PercepCap consistently improves upon the Qwen3-VL baseline and demonstrates leading caption quality.

Problem

• video captioning
• spatio-temporal perception
• perceptual evidence
• object trajectories
• temporal events

Innovation

• structured spatio-temporal perception
• perceive-describe generation chain
• perception-grounded reinforcement learning
• caption-anchored perception data
• explicit perception trace