🤖 AI Summary
To address the high cost of acquiring 3D point cloud data—which limits the scalability of robotic manipulation learning—this paper proposes a vision-based manipulation framework that operates without real 3D inputs. Our core innovation is the 3DStructureFormer module, which transforms monocular RGB images into pseudo-point clouds endowed with explicit geometric structure and employs a dedicated encoder to preserve spatial relationships. We further fuse 2D visual features with these pseudo-3D representations to enable end-to-end policy learning. Evaluated across diverse manipulation tasks—including grasping, pushing/pulling, and insertion—our method achieves performance on par with real-point-cloud baselines, while drastically reducing data acquisition and annotation overhead. Experiments demonstrate that the pseudo-3D representation effectively supports spatial reasoning and generalization. This work establishes a new paradigm for lightweight, deployable vision-manipulation co-learning.
📝 Abstract
Recently,vision-based robotic manipulation has garnered significant attention and witnessed substantial advancements. 2D image-based and 3D point cloud-based policy learning represent two predominant paradigms in the field, with recent studies showing that the latter consistently outperforms the former in terms of both policy performance and generalization, thereby underscoring the value and significance of 3D information. However, 3D point cloud-based approaches face the significant challenge of high data acquisition costs, limiting their scalability and real-world deployment. To address this issue, we propose a novel framework NoReal3D: which introduces the 3DStructureFormer, a learnable 3D perception module capable of transforming monocular images into geometrically meaningful pseudo-point cloud features, effectively fused with the 2D encoder output features. Specially, the generated pseudo-point clouds retain geometric and topological structures so we design a pseudo-point cloud encoder to preserve these properties, making it well-suited for our framework. We also investigate the effectiveness of different feature fusion strategies.Our framework enhances the robot's understanding of 3D spatial structures while completely eliminating the substantial costs associated with 3D point cloud acquisition.Extensive experiments across various tasks validate that our framework can achieve performance comparable to 3D point cloud-based methods, without the actual point cloud data.