The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding

2026-07-27

AI Summary

This work addresses the significant performance degradation of multimodal large language models in spatio-temporal video grounding under sparse-frame inputs, attributed to a mismatch between training and deployment conditions. The study identifies the vision module as the critical bottleneck and demonstrates that fine-tuning only the last three layers of the Vision Transformer (approximately 4% of its parameters) effectively adapts the model to sparse inputs. Furthermore, the authors propose a boundary-aware Hybrid16 frame sampling strategy to enhance temporal boundary localization accuracy. With only 16 input frames, the method achieves a temporal mIoU of 68.8%, surpassing the zero-shot 8B model using dense frames by 12.8 percentage points. Notably, the fine-tuned 2B model consistently outperforms the zero-shot 8B counterpart, highlighting the efficacy of targeted adaptation over sheer scale.

Abstract

Large-scale video platforms process millions of uploads hourly, requiring moderation systems that can localize when and where policy violations occur within each video. Processing every frame is infeasible at scale, so systems are constrained to sparse inputs of 8 to 16 frames per video. Yet state-of-the-art multimodal large language models (MLLMs) are pretrained on dense sequences of hundreds of frames, creating a fundamental mismatch between training and deployment conditions. This mismatch causes severe performance collapse: the Qwen3-VL 8B model drops from 56.0% to 22.3% temporal mIoU when frames are reduced to 16, a 60.2% relative degradation. We present a systematic empirical study of training strategies to close this gap for spatial-temporal video grounding. Our results suggest that visual feature extraction is the dominant bottleneck under sparse-frame inputs. Adapting only the final three ViT layers, 4% of total parameters, achieves 68.8% temporal mIoU and surpasses a zero-shot 8B model using dense inputs by 12.8 points. Language model fine-tuning, by contrast, offers negligible or negative returns. A boundary-aware sampling strategy, Hybrid16, further improves temporal mIoU by 26 points over uniform sampling when temporal boundaries are available. We conclude that for sparse-frame video grounding, training strategy dominates model scale: a fine-tuned 2B model consistently outperforms a zero-shot 8B model, with or without dense frame access.

Problem

• sparse-frame video grounding
• multimodal large language models
• temporal-spatial localization
• visual bottleneck
• video moderation

Innovation

• sparse-frame adaptation
• video grounding
• visual bottleneck
• Hybrid16 sampling
• multimodal LLMs