MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding

2026-08-05

AI Summary

Existing fixed-design video agents struggle to adapt to the shifting distribution of sparse, multimodal evidence in long videos, often leading to redundant computation or degraded performance. This work proposes the first automated agent evolution framework tailored for long-form video understanding. It dynamically assesses information density through sparse frame sampling and query analysis to construct minimally sufficient, independently verifiable tasks. The framework further incorporates Gold Paths trajectory auditing, failure clustering, and modular code updating to enable efficient iterative refinement. Evaluated on VA-EvoBench over four evolution rounds, the approach improves macro-average accuracy from 38.44% to 51.47%, surpassing the current best fixed agent by 6.39 percentage points while using fewer tokens and frames, thereby significantly enhancing both generalization capability and inference efficiency.

Abstract

Long-form video understanding requires locating sparse, question-relevant evidence in long, multimodal videos. Real-world video distributions differ in modality-specific information density, content structure, and evidence patterns, causing fixed video-agent designs to incur redundant processing or fail when mismatched. Extending automated agent evolution from text to video is challenging because full long-video execution makes candidate validation expensive, failures propagate across coupled evidence-processing stages, and complex preprocessing, perception tools, and localization strategies make code-level updates difficult to implement reliably. We introduce MetaVideoAgent, a framework that automatically evolves a video agent for a target distribution. It profiles information density and evidence requirements from sparsely sampled frames and associated queries to guide initial design, then compresses localized failures into independently executable minimal validation tasks. It constructs evidence-grounded Gold Paths, audits Student trajectories, aggregates recurring failures across samples, and attributes them to responsible modules. A modular agent representation constrains each update to the primary responsible module and its necessary dependencies. We further introduce VA-EvoBench, covering eight video distributions with separate evolution and held-out splits. With four evolution iterations per distribution, MetaVideoAgent improves every initial agent and raises macro-average accuracy from 38.44% to 51.47%, at an average evolution cost of 3.54M tokens per distribution. The evolved agents outperform the strongest prior fixed-design video agent by 6.39 percentage points while using the fewest tokens and video frames per question among the compared video agents. We will release all code and data to support reproducible research.

Problem

• long-form video understanding
• video-agent evolution
• evidence localization
• modality-specific information density
• automated agent design

Innovation

• automated agent evolution
• long-form video understanding
• modular agent design
• evidence-grounded validation
• video-agent adaptation