Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

2026-07-17

AI Summary

This work addresses the limitation of existing long-form video question answering methods, which often lack a mechanism to correct early temporal localization errors. To overcome this, the authors propose an iterative self-correcting search framework based on an adaptive temporal tree structure. This framework constructs a non-uniform hierarchical tree guided by visual scene boundaries and explicitly supports exploration and backtracking through four learnable operations: zoom_in, zoom_out, shift, and answer. The model is jointly optimized for both localization accuracy and answer correctness via multi-step trajectory composition, supervised fine-tuning, and reinforcement learning. Experimental results demonstrate significant improvements—12.5 mIoU and 7.4 T-F1 gains on CG-Bench, Haystack-LVBench, and Haystack-Ego4D—and up to 7.1 percentage points higher accuracy on general benchmarks including Video-MME, MLVU, and LVBench.

Abstract

Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these agents typically converge prematurely and cannot recover from an early mistake. We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors. To train this navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards. On three Grounded LVQA benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D), VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D. The learned policy also transfers to general long-video QA, surpassing all prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains: removing either adaptive descent or explicit backtracking substantially degrades performance. Code is available at https://github.com/CeeZh/VTS.

Problem

• Grounded Long Video QA
• temporal backtracking
• evidence localization
• agent-based video search
• self-correction

Innovation

• self-correcting search
• temporal tree
• grounded long-video QA
• agentic navigation
• backtracking