Less from More: Reinforcing Sparse Video Reasoning from Dense References

2026-10-07

AI Summary

This study addresses the unreliable temporal reasoning of video models under sparse-frame conditions by challenging the prevailing assumption that more frames inherently yield greater reliability. To this end, it proposes SAVER, a framework that leverages dense videos as training references and achieves efficient knowledge transfer to sparse frames through reinforced post-training. Specifically, SAVER introduces grounded rewards and reliability-gated reference rewards to optimize the inference process, requiring only minimal temporal localization data without question-answering annotations. Experimental results demonstrate that SAVER consistently improves performance across multiple benchmarks, matching or surpassing dense-frame baselines with substantially fewer frames. Ultimately, this work enables both efficient and reliable sparse video reasoning.

Abstract

Video-language models commonly assume that more temporal observations lead to more reliable reasoning. We question this assumption and argue that the key challenge is not merely processing more video frames efficiently, but learning to reason reliably under limited temporal evidence. We propose SAVER, a dense-to-sparse post-training framework that uses dense video views as training-time references for sparse-frame inference. During reinforcement post-training, paired dense and sparse views are optimized with grounding rewards and a reliability-gated reference reward, encouraging sparse view predictions to preserve task-relevant temporal evidence. Notably, SAVER is trained only on 1,250 randomly sampled temporal grounding examples, without using any video question answering annotations. Across three temporal grounding benchmarks and six video question-answering benchmarks, SAVER consistently improves performance across frame budgets. In particular, SAVER can match or surpass dense-frame Qwen3.5 baselines while using substantially fewer frames. These results show that temporal grounding can serve as an effective evidence-localization proxy for learning sparse video reasoning that transfers to broader video understanding tasks.

Problem

• sparse video reasoning
• temporal evidence
• video-language models
• video understanding

Innovation

• Sparse Video Reasoning
• Dense-to-Sparse Post-training
• Reinforcement Learning
• Temporal Grounding
• Reliability-Gated Reference Reward