GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience

2026-08-03

AI Summary

Existing video memory systems struggle to simultaneously support query-based retrospective retrieval and proactive assistance tailored to current contexts, often lacking a unified memory architecture. This work proposes a training-free, causal incremental memory framework that organizes memories hierarchically across time, progressively aggregating fine-grained perceptual evidence into representations at the levels of moments, events, and cross-day patterns. It introduces a scale-native multi-scale retrieval mechanism that seamlessly unifies reactive question answering with proactive contextual support. Evaluated on benchmarks such as MM-lifelong and EgoServe, the framework achieves state-of-the-art performance. Ablation studies confirm the complementary roles of each temporal hierarchy and the efficacy of the multi-scale retrieval design, with particularly notable gains in cross-day reasoning scenarios.

Abstract

A wearable assistant should both answer questions about its visual history and recognize when that history is useful to the present situation. Existing video-memory systems primarily support question-conditioned recall, whereas proactive assistants typically use separate memory and control mechanisms. We introduce GROVE, a training-free framework that supports both behaviors with one memory grown causally from a continuous video stream. GROVE retains fine-grained perceptual evidence and incrementally consolidates it into time-stamped moments, coherent episodes, and recurring cross-day patterns. Each stratum is paired with a scale-native retrieval skill for locating an observation, replaying an activity, or traversing long-range regularities. Reactive QA and proactive assistance share this memory and access interface, differing in whether retrieval is initiated by a user query or the current situation. Across multiple benchmarks including the challenging MM-lifelong and EgoServe, GROVE achieves the best results among the compared methods. Controlled ablations show that the temporal strata and their access skills are complementary, with patterns providing the largest benefit when evidence spans multiple days. Code will be available at https://github.com/SitongGong/GROVE.

Problem

• video memory
• temporal reasoning
• proactive assistance
• reactive QA
• lifelong learning

Innovation

• temporally stratified memory
• training-free framework
• streaming video experience
• proactive assistance
• memory consolidation