EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision

2026-05-29

AI Summary

Existing streaming video benchmarks struggle to enable fine-grained evaluation of embodied agents’ episodic memory capabilities, particularly lacking controllable diagnostics for both memory content and retention duration. This work proposes the first first-person visual streaming benchmark for episodic memory, comprising 2,250 questions organized along seven cognitive dimensions and 8,528 recall conditions. It introduces an Answer Validity Window (AVW) mechanism to disentangle model forgetting from environmental changes. Leveraging a unified Qwen3-VL multimodal framework, the study systematically compares multiple memory management strategies, revealing heterogeneous performance in detail preservation, temporal structure maintenance, and ultra-long-term recall. Experiments demonstrate that current approaches all fail to achieve real-time processing (>1 second per frame), with peak accuracy reaching only approximately 45%, thereby exposing critical limitations in existing architectures.

Abstract

Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchmarks provide limited tools for diagnosing what models remember and for how long. We introduce \egostream, a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision. \egostream organizes 2,250 curated questions along seven cognitive dimensions: detail, spatial, temporal, event, social, causal, and prospective memory. We introduce the Answer Validity Window (AVW), which specifies the temporal span an answer remains valid as the observed scene evolves. This allows us to expand the questions into 8,528 recall-conditioned evaluations, enabling controlled testing from instant to ultra-long-term recall while separating genuine model forgetting from natural world-state changes. We rigorously establish baseline performance through a unified streaming MLLM framework that compares several state-of-the-art memory-management mechanisms, covering sliding windows, attention sinks, KV-cache pruning, merging, and offloading. Experiments within a unified Qwen3-VL backbone reveal that comparable aggregate accuracies mask starkly different memory profiles. For instance, token pruning preserves fine-grained details and temporal structure significantly better than token merging, while quantized offloading rescues ultra-long-term recall. Ultimately, all mechanisms operate well below real-time (>1s per frame), and top performing methods ceil at about 45\% accuracy, exposing critical gaps in current architectures. \egostream provides the diagnostic testbed needed to close these gaps.

Problem

• streaming episodic memory
• egocentric vision
• memory benchmark
• autonomous agents
• forgetting diagnosis

Innovation

• episodic memory
• egocentric vision
• streaming benchmark
• Answer Validity Window
• memory management