MEMORA: Embodied Action Memory from Egocentric Videos for Reasoning and Planning

2026-07-15

AI Summary

Existing long-horizon robotic planning approaches lack an effective memory mechanism for embodied experience, hindering their ability to comprehend future goals and reuse past behavioral patterns. This work proposes an Embodied Action Memory (EAM) system that, for the first time, integrates four types of memory—environmental, entity, activity, and inferential knowledge—into a unified “formation–consolidation–retrieval” memory lifecycle. The system maintains object identities and state histories through online editing, abstracts recurring experiences into reusable programs offline, and leverages language models to enable memory-based reasoning and planning. Experiments on MEMORA-Bench demonstrate that the proposed method significantly outperforms baseline approaches, achieving up to a 20.5% improvement in memory evaluation accuracy and a 16.6% relative gain in out-of-distribution task planning scores.

Abstract

Long-horizon robot planning requires more than predicting what actions will do next; it also requires memory of the embodied experience that makes future goals interpretable. People do not plan from the present scene alone: they draw on remembered places, object-state changes, prior procedures, and regularities revealed through repeated action. We formulate Embodied Action Memory (EAM) as the capability to form, maintain, and use such experience as a persistent memory state for later decisions. MEMORA realizes EAM with a formation-consolidation-retrieval lifecycle and four typed stores: Environment Memory, Entity Memory, Activity Memory, and Inferred Knowledge. Online editing maintains object identities and state histories as new observations arrive; offline consolidation abstracts repeated experience into reusable procedures and participant-specific regularities. MEMORA-Bench evaluates this lifecycle on 45 hours of EPIC-KITCHENS-100 extension video across 18 participants through memory-grounded planning, including previously unseen goals, and a complementary memory-assessment task. Across four open-weight language models, full MEMORA--combining editing, typed stores, and consolidation--achieves the strongest aggregate results among the evaluated memory conditions. It improves memory-assessment accuracy by up to 20.5 points over the strongest controlled baseline and improves out-of-distribution Robot-Grounded Plan score by up to 16.6% relative. A qualitative two-task robot deployment study further illustrates how memory-grounded language plans can interface with downstream control, while the overall results show that editable, consolidated memory can supply remembered context for robot planning. Project page: https://yuzihaowashu.github.io/MEMORA/

Problem

• Embodied Action Memory
• long-horizon planning
• memory-grounded reasoning
• egocentric video
• robot planning

Innovation

• Embodied Action Memory
• Memory Lifecycle
• Typed Memory Stores
• Online Editing
• Offline Consolidation