Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation

2026-09-29

AI Summary

This study addresses the challenge of unreliable execution in partially observable robotic manipulation caused by latent states. To this end, it proposes the SEEK framework and the HIDE benchmark. SEEK integrates three distinct memory mechanisms to retain historical evidence and track execution states. As the first evaluation benchmark for skill-level memory, HIDE leverages POMDP modeling and multimodal memory augmentation to systematically demonstrate that synergistic multi-mechanism coordination outperforms single-strategy approaches. Experimental results indicate that the proposed method significantly improves task success rates in both simulated and real-world environments, underscoring the critical role of precisely aligning memory design with task-specific information requirements.

Abstract

Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However, reliable execution often depends on hidden task states that cannot be determined from current observations alone, making interaction history essential. We introduce $HIDE$, a benchmark for evaluating manipulation memory under partial observability. HIDE comprises 15 tasks covering repetition counting, historical-state recall, and execution-progress tracking, with randomized initial configurations and decision points where similar observations require different actions depending on prior events. We further propose $SEEK$, a framework combining three complementary memory mechanisms to retain historical evidence and track execution state. Evaluations reveal substantial limitations in existing policies on HIDE, while memory augmentation improves task success in both simulation and real-world experiments. Individual mechanisms benefit some tasks but can degrade others; their combination achieves the highest average success rate on HIDE among the evaluated configurations. These findings highlight the importance of maintaining internal representations of hidden task states and matching memory design to task-specific information requirements.

Problem

• partially observable robotic manipulation
• skill-level memory
• hidden task states
• benchmarking
• interaction history

Innovation

• Partially Observable Robotic Manipulation
• Skill-Level Memory
• Benchmarking
• Memory Augmentation
• Hidden Task States