SkillMemo: Expert-guided Skill Memory Framework for Compositional Embodied Manipulation

2026-08-06

AI Summary

Existing embodied visuomotor models struggle to achieve compositional generalization in out-of-distribution scenarios and fail to effectively model reusable skill structures, largely due to the scarcity of large-scale trajectory data. This work proposes SkillMemo, a novel framework that uniquely integrates expert-guided Mixture-of-Experts (MoE) architecture with a skill-level retrievable memory mechanism. By leveraging MoE to implicitly segment long-horizon demonstrations into atomic skills, SkillMemo constructs a skill-level key-value episodic memory bank. During inference, relevant skills are retrieved and fused through gating to enhance action prediction. The approach substantially improves compositional generalization and achieves state-of-the-art performance in both simulated and real-world tasks, significantly outperforming the π₀.₅ baseline.

Abstract

Embodied visuomotor models, including Diffusion Policy (DP) and Vision-Language-Action (VLA) models, have demonstrated promising performance on robotic manipulation benchmarks. However, their potential remains fundamentally constrained by the scarcity of large-scale embodied trajectory datasets, leading to insufficient compositional generalization in out-of-distribution (OOD) scenarios with limited capability to capture reusable skill structures. To address this limitation, we propose Skill-Based Memory (SkillMemo) framework that implicitly decomposes long-horizon demonstrations into latent atomic skills and integrates skill-level features into a dynamic episodic memory bank for solving compositional tasks. Specifically, we first introduce an expert-guided trajectory segmentation module built upon a Mixture-of-Experts (MoE) architecture, which implicitly partitions trajectories into distinct skill primitives represented by learned gating coefficients. We further design a skill-level episodic memory architecture that stores compact skill representations as retrievable key-value pairs. During inference, the memory bank retrieves the most relevant skill primitives which are subsequently fused with the model's current gating distribution, providing a robust contextual prior to refine action predictions. Extensive experiments on the simulation benchmark and real-world manipulation tasks demonstrate that SkillMemo consistently enhances both DP and VLA backbones, achieving state-of-the-art performance and outperforming $π_{0.5}$, while exhibiting strong compositional generalization to unseen task configurations.

Problem

• compositional generalization
• embodied manipulation
• skill decomposition
• out-of-distribution
• trajectory scarcity

Innovation

• SkillMemo
• compositional generalization
• Mixture-of-Experts
• episodic memory
• embodied manipulation