🤖 AI Summary
Existing video frame sampling evaluation methods struggle to effectively measure how well sampled frames capture the informativeness and representativeness of a video. To address this limitation, this work proposes STEC, a novel no-reference evaluation metric that integrates spatiotemporal structural entropy with coverage in a lightweight, task-agnostic manner without requiring access to the original reference video. STEC quantifies sampling quality by modeling three key aspects: the spatial information strength of individual frames, the temporal distribution breadth across the sequence, and the non-redundant coverage of visual content. Experiments on MSR-VTT test-1k demonstrate that STEC effectively discriminates among random, uniform, and content-aware sampling strategies and reveals their robustness differences at the individual video level.
📝 Abstract
Frame sampling is a fundamental component in video understanding and video--language model pipelines, yet evaluating the quality of sampled frames remains challenging. Existing evaluation metrics primarily focus on perceptual quality or reconstruction fidelity, and are not designed to assess whether a set of sampled frames adequately captures informative and representative video content. We propose Spatio-Temporal Entropy Coverage (STEC), a simple and non-reference metric for evaluating the effectiveness of video frame sampling. STEC builds upon Spatio-Temporal Frame Entropy (STFE), which measures per-frame spatial information via entropy-based structural complexity, and evaluates sampled frames based on their temporal coverage and redundancy. By jointly modeling spatial information strength, temporal dispersion, and non-redundancy, STEC provides a principled and lightweight measure of sampling quality. Experiments on the MSR-VTT test-1k benchmark demonstrate that STEC clearly differentiates common sampling strategies, including random, uniform, and content-aware methods. We further show that STEC reveals robustness patterns across individual videos that are not captured by average performance alone, highlighting its practical value as a general-purpose evaluation tool for efficient video understanding. We emphasize that STEC is not designed to predict downstream task accuracy, but to provide a task-agnostic diagnostic signal for analyzing frame sampling behavior under constrained budgets.