Video-Index: A Curated Meta-Benchmark for Video Understanding

2026-09-30

AI Summary

This study addresses the issue of models exploiting shortcuts rather than genuinely comprehending content in video benchmarks by proposing a five-layer attack pyramid auditing framework. Methodologically, it introduces a novel agent-based deterministic benchmark composition pipeline, integrating adversarial auditing, near-duplicate detection, and red-team gating techniques for rigorous data curation. Based on this framework, we construct and release Video-Index, a shortcut-resistant meta-benchmark for video understanding comprising 840 high-difficulty validation items. Evaluations demonstrate that this benchmark effectively reveals the true comprehension capabilities of models, uncovering that Claude Opus 5 significantly outperforms existing open-source models.

Abstract

A video benchmark should reward the capability it claims to measure, yet models can exploit answer options, question text, or partial visual evidence. We introduce the attack pyramid, five levels of shortcut attacks with increasing access to each item, and audit 115 video benchmarks with it. On 35 benchmarks, attackers that never see a frame approach full-video accuracy. On 51 benchmarks with temporal probes, shuffled frames keep a median 96% of full-video accuracy. Near-duplicate questions make up at least half the items in 63 benchmarks. We screen 505,518 question-answer pairs from 112 of them into an audited pool. Agents turn evaluation requests into specifications, and a deterministic selector with a red-team gate composes reproducible benchmarks. We release Video-Index, the 210 hardest verified items under these attacks in each of four capability groups, 840 items from 76 sources. With the same fixed input, Claude Opus 5 outscores every open-source model by over 37 percentage points, and agent tools add about 20 more, yet all systems leave room to improve efficiency and accuracy. Blog: https://www.enxinsong.com/blog/video-index/ GitHub: https://github.com/Espere-1119-Song/Video-Index Hugging Face: https://huggingface.co/datasets/Video-Index/Video-Index

Problem

• Video Understanding
• Benchmark Evaluation
• Shortcut Attacks
• Dataset Auditing

Innovation

• Meta-Benchmark
• Attack Pyramid
• Shortcut Attacks
• Red-team Gate
• Video Understanding