ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding

2026-07-30

AI Summary

This work addresses the challenge in streaming video understanding where future frames are unknown and critical visual evidence must be continuously retained, a setting in which existing methods lack explicit modeling of temporally evolving objects. The paper proposes a training-free framework that, for the first time, leverages latent objects as spatially consistent memory anchors to construct persistent object trajectories by clustering and cross-frame association from frozen video large language model representations. Within limited memory, it effectively fuses object history, instantaneous changes, and local context—without requiring external detectors or architectural modifications—to enable robust reasoning about object identity, interactions, and states. Evaluated on OVO-Bench, the approach boosts Qwen2.5-VL-7B by 10.0 points while reducing peak GPU memory and first-token latency by approximately 50%; on long-video tasks, it surpasses full-token baselines using only 17.5% of visual tokens.

Abstract

Streaming video understanding requires models to continuously retain useful visual evidence before future questions are known. Existing approaches primarily manage the growing visual context according to token importance, temporal redundancy, or segment-level relevance, but rarely organize evidence around objects that persist and evolve over time. Thus, in this paper, we introduce ObjectStream, a training-free framework that treats latent objects as memory anchors for streaming video understanding. ObjectStream induces spatially coherent latent objects directly from frozen Video-LLM representations, links them across frames into persistent anchors, and maintains their histories under a bounded memory budget, without requiring external object detectors or segmentation models. Built on these anchors, ObjectStream preserves three complementary forms of evidence: persistent object histories, transient object changes, and recent visual context. This design enables existing Video Large Language Models (Video-LLMs) to reason over object identities, interactions, and state changes while leaving the underlying model unchanged. Extensive experiments on online streaming and offline long-video benchmarks demonstrate both effectiveness and efficiency. In online streaming evaluation, ObjectStream improves Qwen2.5-VL-7B by 10.0 points on OVO-Bench Real-Time Visual Perception, while reducing peak GPU mem-ory and TTFT by approximately 50%. On offline long-video benchmarks, it surpasses the full-token baseline while discarding 82.5% of visual tokens. These results highlight latent objects as a practical and effective organizing principle for compact streaming video memory.

Problem

• streaming video understanding
• memory management
• latent objects
• video evidence retention
• object persistence

Innovation

• latent objects
• memory anchors
• streaming video understanding
• Video-LLM
• training-free framework