StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

2026-03-22

AI Summary

Current large video models lack a systematic evaluation of the trade-offs among efficiency, memory usage, and accuracy in streaming understanding scenarios. This work proposes StreamingEval, the first unified benchmarking framework designed for real-world resource constraints. StreamingEval standardizes historical visual context through a fixed-capacity memory buffer and jointly assesses visual encoding efficiency, text decoding latency, and task performance, while introducing quantifiable deployability metrics. Experimental results demonstrate that existing models struggle to meet the demands of practical streaming applications. This study establishes a comprehensive benchmark and provides clear directions for future optimization in streaming video understanding.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at https://github.com/wwgTang-111/StreamingEval1.

Problem

• streaming video understanding
• Video-LLMs
• evaluation protocol
• realistic constraints
• system deployability

Innovation

• StreamingEval
• Video-LLM
• realistic evaluation
• memory-constrained streaming
• efficiency-accuracy trade-off