Technology Category
Intelligent Robots: Multimodal Perception & Sensor FusionComputer Vision: Diffusion Models for VisionNatural Language Processing: Safety and Robustness
Self-Modeling Robots by Photographing
🤖 AI Summary
Existing robot self-modeling approaches suffer from limitations in modeling accuracy, texture representation capability, and data efficiency—particularly lacking exploration of link-level texture-aware modeling. This paper proposes the first link-level, texture-aware, end-to-end differentiable robot self-modeling framework: given only multi-view RGB images—without depth sensors or prior structural knowledge—it jointly reconstructs geometry, kinematic parameters, and surface texture. Our core innovation lies in the first integration of 3D Gaussian Splatting with neural ellipsoidal skeletal representations, augmented by multi-view geometric supervision and Gaussian-clustering-driven deformable skeletal construction. The method drastically reduces data acquisition cost, enables high-fidelity novel-view synthesis, and significantly improves downstream task performance—including inverse kinematics solving and motion planning—in both accuracy and generalization. It outperforms state-of-the-art self-modeling methods across multiple quantitative metrics.
Technology Category
Application Category
📝 Abstract
Self-modeling enables robots to build task-agnostic models of their morphology and kinematics based on data that can be automatically collected, with minimal human intervention and prior information, thereby enhancing machine intelligence. Recent research has highlighted the potential of data-driven technology in modeling the morphology and kinematics of robots. However, existing self-modeling methods suffer from either low modeling quality or excessive data acquisition costs. Beyond morphology and kinematics, texture is also a crucial component of robots, which is challenging to model and remains unexplored. In this work, a high-quality, texture-aware, and link-level method is proposed for robot self-modeling. We utilize three-dimensional (3D) Gaussians to represent the static morphology and texture of robots, and cluster the 3D Gaussians to construct neural ellipsoid bones, whose deformations are controlled by the transformation matrices generated by a kinematic neural network. The 3D Gaussians and kinematic neural network are trained using data pairs composed of joint angles, camera parameters and multi-view images without depth information. By feeding the kinematic neural network with joint angles, we can utilize the well-trained model to describe the corresponding morphology, kinematics and texture of robots at the link level, and render robot images from different perspectives with the aid of 3D Gaussian splatting. Furthermore, we demonstrate that the established model can be exploited to perform downstream tasks such as motion planning and inverse kinematics.
Problem
Research questions and friction points this paper is trying to address.
Develops a high-quality, texture-aware robot self-modeling method.
Addresses low modeling quality and high data acquisition costs.
Enables robot morphology, kinematics, and texture modeling without depth data.
Innovation
Methods, ideas, or system contributions that make the work stand out.
3D Gaussians represent robot morphology and texture
Neural ellipsoid bones controlled by kinematic network
Training with joint angles and multi-view images
🔎 Similar Papers
No similar papers found.
