RORA: Realistic Object Reconstruction with Articulation

2026-08-05

AI Summary

This work addresses the challenge of reconstructing articulated 3D assets with joint-based motion from real-world objects, a task where existing methods often rely on complex dynamic scans and suffer from low success rates. The authors propose an end-to-end pipeline that requires only a single static video of the object, augmented with minimal human-in-the-loop interaction, to produce high-fidelity, physically interactive articulated models. A key innovation is the first method capable of automatically inferring plausible joint axes directly from static video, enabling efficient user validation. The approach combines a hybrid representation of 3D Gaussian splatting and mesh geometry, integrated with convex decomposition and user-guided part grouping. Experiments demonstrate high-accuracy articulation reconstruction on both the PartNet-Mobility-v0 benchmark and real-world objects, with successful deployment in dexterous manipulation tasks within Unreal Engine and Isaac Sim.

Abstract

Replicating real-world environments into simulation by realistic visual representation like NeRF and 3D Gaussian Splatting (3DGS) has emerged as an effective strategy to reduce the sim-to-real gap in robot learning. However, implementing object articulation during the real-to-sim process is still a challenging task. Existing motion tracking or learning based articulation methods shows low success rates on complex kinematic structures having multiple joints. Furthermore, those methods require scan of dynamic motion of objects, which makes reconstruction process much complicated. In this work, we propose the first end-to-end pipeline that reconstructs simulation-ready assets with accurate articulation from a single static object video input through suggestion based human-in-the-loop process. Our approach exports a hybrid representation combining 3DGS for photorealistic rendering and mesh-based geometry for physical interaction. In the reconstruction process, our pipeline performs convex decomposition followed by user grouping for intuitive part segmentation, subsequently binding 3D Gaussians to the corresponding mesh parts. An Automatic Joint Suggestion Algorithm then calculates candidate joint axes from local boundary geometries and presents them to users for efficient articulated asset reconstruction. We have shown that our method achieves precise articulation results on partnet-mobility-v0 dataset and real objects. Additionally we presented a potential usage of our framework on robot learning, deploying the reconstructed assets in Unreal Engine and NVIDIA Isaac Sim, demonstrating real-time dexterous hand manipulation tasks.

Problem

• object articulation
• real-to-sim
• 3D reconstruction
• sim-to-real gap
• articulated objects

Innovation

• articulated object reconstruction
• 3D Gaussian Splatting
• human-in-the-loop
• convex decomposition
• joint axis suggestion