MemER: Scaling Up Memory for Robot Control via Experience Retrieval

2025-10-23

AI Summary

Existing robot policies struggle to balance efficiency and robustness in long-term memory: sequence-based modeling incurs high computational overhead and is vulnerable to covariate shift, while naive history subsampling leads to either information redundancy or loss of critical frames. This paper proposes a hierarchical memory-augmented framework: a high-level vision-language policy (Qwen2.5-VL-7B-Instruct) actively retrieves task-relevant historical keyframes, and a low-level action policy (π₀.₅) executes fine-grained manipulation conditioned on them—enabling minute-scale temporal dependency modeling. Its core innovation is an experience-driven keyframe retrieval mechanism that drastically reduces historical processing burden and enhances cross-temporal reasoning. Evaluated on three real-world long-horizon robotic manipulation tasks, our method outperforms state-of-the-art approaches, validating a lightweight, robust, and deployable paradigm for memory extension.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Image and Video Retrieval
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Humans routinely rely on memory to perform tasks, yet most robot policies lack this capability; our goal is to endow robot policies with the same ability. Naively conditioning on long observation histories is computationally expensive and brittle under covariate shift, while indiscriminate subsampling of history leads to irrelevant or redundant information. We propose a hierarchical policy framework, where the high-level policy is trained to select and track previous relevant keyframes from its experience. The high-level policy uses selected keyframes and the most recent frames when generating text instructions for a low-level policy to execute. This design is compatible with existing vision-language-action (VLA) models and enables the system to efficiently reason over long-horizon dependencies. In our experiments, we finetune Qwen2.5-VL-7B-Instruct and $π_{0.5}$ as the high-level and low-level policies respectively, using demonstrations supplemented with minimal language annotations. Our approach, MemER, outperforms prior methods on three real-world long-horizon robotic manipulation tasks that require minutes of memory. Videos and code can be found at https://jen-pan.github.io/memer/.

Problem

• Enabling robot policies with memory for task performance
• Addressing computational cost and brittleness of long observation histories
• Selecting relevant keyframes for efficient long-horizon reasoning

Innovation

• Hierarchical policy framework selects relevant keyframes
• High-level policy generates instructions using retrieved memories
• Compatible with existing vision-language-action models