Problem
Research questions and friction points this paper is trying to address.
visuomotor learning
privileged 3D supervision
geometric representation
robot manipulation
LEAP: Making Privileged Geometry Supervision Effective for Visuomotor Learning
🤖 AI Summary
This study addresses the limitation that visual representations learned under privileged 3D supervision often lack geometric information essential for policy control and are prone to proprioceptive shortcuts and dominant-view dependencies. To mitigate these issues, this work proposes the LEAP framework, which optimizes the visual encoder through workspace point cloud reconstruction, wrist-view dropout, and a local geometric alignment mechanism. It introduces privileged geometric supervision during training while relying solely on RGB and proprioceptive signals at inference. This approach significantly enhances policy robustness without incurring additional inference overhead, substantially outperforming baseline models such as Diffusion Policy on benchmarks including RoboTwin.
📝 Abstract
Privileged 3D supervision uses additional geometric information during training to guide RGB-based visuomotor policy learning, without requiring geometric inputs at deployment. However, low reconstruction error does not ensure that visual representations capture geometry useful for control. We identify three limitations that weaken this supervision: proprioceptive shortcuts, dominant-view reliance, and reconstruction objectives dominated by task-irrelevant geometry. To address these limitations, we propose Latent Encoding with Aligned Privileged Geometry (LEAP). Our framework uses an auxiliary decoder to reconstruct point clouds within the manipulation workspace from visual features alone, while retaining proprioception for action prediction. Alongside full reconstruction, we introduce wrist-view dropout and partial reconstruction targets matched to the retained wrist, encouraging the encoder to capture complementary local geometry. The auxiliary decoder is removed at inference, leaving only RGB observations and robot state as policy inputs. Extensive experiments on RoboTwin, ManiSkill, and real-world tasks demonstrate consistent and substantial improvements over Diffusion Policy and ACT, with only a small increase in parameter count and no reduction in inference speed.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Privileged Geometry Supervision
Visuomotor Learning
Point Cloud Reconstruction
Wrist-view Dropout
Auxiliary Decoder
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
No related jobs found.
