VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation

2026-06-06

AI Summary

Existing long-form video generation methods rely on handcrafted pipelines and lack the capacity for adaptive skill evolution. This work proposes the first agent-based evaluation and skill evolution framework tailored for long video generation, enabling agents to autonomously decompose high-level instructions into executable workflows of primitive skills. The framework integrates multimodal foundation models, execution trajectory analysis, and an agent-as-judge evaluation mechanism to support automatic skill optimization and composition. Experiments demonstrate that explicit skill composition significantly outperforms single-skill approaches, that skill evolution effectively enhances video quality, and that the proposed evaluation metrics align closely with human judgments—particularly along procedural dimensions.

Abstract

Recent agent frameworks such as Claude Code, Codex, and OpenClaw are strong at tool use and orchestration, but whether they can handle long video generation, a long-horizon multimodal task, remains underexplored. Unlike earlier video agents whose pipeline is handcrafted, these frameworks can build and refine their own workflows. We introduce VideoWeaver, an agent harness and benchmark that evaluates and evolves skills for long video generation, where an agent turns a single instruction into a long video by composing foundation skills into its own workflow rather than following a predefined pipeline. The benchmark has 16 task categories and 285 cases, with references spanning text, image, audio, video, and their combinations. Because errors can arise at any stage and not just in the final video, we propose an agent-as-judge that inspects both the execution trace and the final video, grounding its scores in evidence such as metadata and intermediate files. Using this feedback, we further design a skill evolution algorithm that refines and merges the agent's skills. Across multiple frameworks and models, we find that an explicit composition skill improves the generation process over using foundation skills alone, that skill evolution further improves output quality, and that performance varies notably across harness and model choices. The proposed agent-as-judge also aligns well with human judgments, especially on process metrics. Code and dataset is available at https://github.com/JianhuiWei7/VideoWeaver

Problem

• long video generation
• agentic frameworks
• multimodal tasks
• skill composition
• agent evaluation

Innovation

• agentic video generation
• skill composition
• agent-as-judge
• skill evolution
• long-horizon multimodal tasks