VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing

2026-08-05

AI Summary

Existing evaluation methods for video generation and editing are constrained by fixed content, limited scene coverage, and a lack of interpretable scoring rationales. This work proposes VideoArgus, the first unified, reusable, and evidence-driven automatic evaluation framework encompassing five video-related tasks. For each input sample, VideoArgus automatically generates fine-grained, output-agnostic scoring criteria and leverages visual-language model–based question answering, visual analysis tools, and relevance verification to produce scores, justifications, and diagnostic reports grounded in explicit evidence. Evaluated on a human-aligned test set of 1,260 videos, VideoArgus achieves higher Spearman and Kendall correlation coefficients than task-specific evaluators, with stable ranking performance across different backbone models.

Abstract

Evaluating generated videos remains challenging because existing benchmarks rely on fixed evaluation content, cover only a subset of generation and editing settings, and provide limited evidence for their scores. We introduce VideoArgus, a unified rubric-grounded framework covering five video generation and editing settings. For each input instance, VideoArgus generates an output-blind, sample-specific rubric once and reuses it to evaluate all corresponding candidate videos. The rubric defines concrete criteria, scoring rules, failure modes, and evidence plans, which guide criterion-specific VLM QA and visual tools to produce evidence-grounded criterion scores, rationales, and a diagnostic report. We further construct VideoArgus-Bench, containing 1,026 curated input instances built from 653 high-quality images and 416 high-quality videos, with all benchmark rubrics pre-generated, frozen, and released. On a separate 1,260-video human-alignment set, VideoArgus achieves higher within-input Spearman and Kendall correlations with human judgments than the corresponding benchmark-specific evaluators across all five tasks. Model rankings also remain largely consistent across different rubric-generation and evaluation-VLM backbones. All code and data are released. Visit our project page: https://zzzmyyzeng.github.io/VideoArgus

Problem

• video evaluation
• rubric-based assessment
• video generation
• video editing
• benchmarking

Innovation

• rubric-grounded evaluation
• video generation
• video editing
• visual language models
• unified benchmarking