Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

2026-08-07

AI Summary

This work addresses a critical yet overlooked issue in video large language models (Video-LLMs): despite overall performance gains with increased visual budgets, a non-monotonic dependency exists at the instance level, where 12.5%–25.5% of samples exhibit degraded performance under higher frame counts—a phenomenon this study terms “high-budget degradation.” To diagnose and mitigate this effect, the authors introduce novel techniques including match-grid metrics, structured frame auditing, and confidence-cascaded reasoning. Through instance-level trajectory modeling and controlled ablation studies, the proposed approach successfully recovers performance in 29.0% of degraded cases while reducing average frame consumption by 31.7% without compromising accuracy.

Abstract

Aggregate scaling curves suggest that Video LLMs improve smoothly or saturate as visual budgets grow. We show that this view can conceal large, opposing changes at the item level. We represent each frozen model--item pair by its response trajectory under controlled visual budgets and derive matched-grid measures of configuration complementarity, harmful transitions, and text overwrite. Across five open Video LLMs from three architecture families, four multiple-choice benchmark splits, open-ended QA and summarization, and fixed-history dialogue generation, no single budget serves all items. On the four-model matched MCQA grid, item-level oracle headroom spans $8.8$--$18.9$ accuracy points and $12.5$--$25.5\%$ of items are correct at a lower budget but wrong at a higher one. Task-appropriate continuous metrics show the same complementarity beyond multiple choice: Token-F1 oracle gaps are $2.7$--$3.7$ score points on MLVU generation and $3.8$--$4.8$ points on AVSD current-turn generation, even when mean quality improves with budget. The effect persists across frame count, spatial resolution, sampling policy, temporal--spatial allocation, and independently executed raw-video and cached pipelines, with per-item rates and membership tracking protocol choices. A controlled sampling intervention recovers $29.0\%$ of terminal regressions, and a structured frame audit identifies several recurring evidence pathways. We release per-item trajectories, protocol provenance, derived annotations, and reproducible analysis code as an auditing artifact. A confidence cascade matches fixed-$128f$ accuracy while reducing average shared frame cost by $31.7\%$, illustrating one operational use of the response matrix.

Problem

• Video LLMs
• scaling heterogeneity
• item-level performance
• visual budget
• performance regression

Innovation

• item-level scaling heterogeneity
• response trajectory
• visual budget
• configuration complementarity
• confidence cascade