StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding

2026-08-06

AI Summary

Existing video understanding methods struggle with hour-long continuous audiovisual streams due to insufficient long-term memory, limited real-world interactivity, and evaluation protocols confined to short clips and multiple-choice questions. To address these limitations, this work introduces StreamArena, a benchmark comprising 243 long-form videos and 3,646 open-ended question-answer pairs designed to evaluate capabilities in real-time perception, historical recall, active interaction, and multimodal tool usage. The authors further propose StreamMind, a two-tier architecture featuring a front-end independent scheduler for low-latency interactive responses and a back-end asynchronous module that constructs persistent multimodal memory to support historical reasoning and external retrieval. Experiments demonstrate that StreamMind significantly outperforms existing streaming baselines across all four core capabilities and achieves substantially reduced response latency through reuse of persistent internal states.

Abstract

Deploying autonomous multimodal agents in continuous, real-world environments requires them to ingest unbounded audio-visual streams and maintain hour-scale memory. However, current evaluations predominantly rely on brief clips and multiple-choice formats. This design allows minimal baselines that process only the last four frames to match or surpass complex streaming models, while answer options also expose language shortcuts. We introduce StreamArena, a benchmark for hour-scale, interactive streaming video understanding. StreamArena contains 243 full-length videos averaging 88.8 minutes and 3,646 rigorously annotated, open-ended question-answer pairs that evaluate real-time perception, historical retrospection, proactive interaction, and multimodal tool utilization. Evaluation across diverse systems exposes a tension between continuous interaction and long-horizon multimodal comprehension. Methods that retain only recent frames cannot recover distant events, methods that convert past observations into text lose visual evidence, and methods that repeatedly compress visual memory struggle to preserve fine-grained details over time. We address this tension with StreamMind, a two-tier architecture that assigns latency-critical interaction and proactive monitoring to independently scheduled frontend workers, while backend workers asynchronously construct persistent multimodal memory and perform historical recall and external search. StreamMind outperforms existing streaming baselines across all four capabilities and reduces query-to-answer latency by reusing persistent state.

Problem

• streaming video understanding
• long-horizon comprehension
• multimodal agents
• continuous interaction
• hour-scale memory

Innovation

• streaming video understanding
• long-horizon memory
• multimodal agents
• interactive perception
• persistent memory architecture