CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding

2026-09-30

AI Summary

This study addresses the challenge of balancing long-range search with fine-grained understanding under limited visual budgets in very long video temporal localization. We propose a strategy-tool co-evolution framework that jointly evolves high-level strategies and media tools from VLM reasoning trajectories into reusable skills, autonomously coordinating multimodal observations for intelligent reasoning. Without updating model parameters, this approach employs an external skill updater to distill task experience and upgrade code-based tools, thereby eliminating reliance on strong planning models. Evaluated across five benchmarks and three VLMs, our method significantly improves localization accuracy while reducing visual token costs, and demonstrates strong generalization to general long-form video question answering.

Abstract

Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.

Problem

• Ultra-long Video Temporal Grounding
• Agentic Methods
• Visual Budget
• Policy-Tool Coevolution
• Long-video Understanding

Innovation

• Policy-Tool Coevolution
• Ultra-Long Video Temporal Grounding
• Agentic Reasoning
• Skill Updater
• Visual Language Model