USDCraft: Geometrically Grounded Programmatic Modeling of Articulated 3D Assets for Simulation

2026-10-08

AI Summary

This study addresses the challenge of reconstructing articulated 3D assets, which is often hindered by out-of-distribution shifts in real-world objects or incomplete meshes. To overcome this, the authors propose a training-free procedural modeling framework driven by large language models (LLMs). Specifically, the LLM generates executable programs that automatically construct simulation-ready articulated assets from partial geometric evidence. Physical attributes are subsequently populated through iterative geometric verification and visual feedback, with the final outputs exported in USD format for compatibility with Isaac Sim. The proposed approach achieves state-of-the-art articulation recovery rates across two benchmarks. Furthermore, the study validates the effectiveness of the real-to-sim-to-real closed-loop pipeline in robotic manipulation tasks, demonstrating its practical utility for embodied AI applications.

Abstract

Geometrically faithful and functional articulated 3D assets are essential for real-to-sim robot manipulation, where policies trained in simulation must transfer to physical objects. Recent mesh-based methods learn to infer articulation from annotated 3D assets, but deployment remains challenging when real-world objects fall outside the training distribution or their meshes are incomplete or corrupted. To address these limitations, we formulate articulated asset reconstruction as programmatic modeling grounded in partial geometric evidence and introduce USDCraft, a framework in which a pretrained LLM writes and revises executable programs for simulation-ready articulated assets without task-specific training. We propose source geometry analysis, which converts the source mesh into a metric textual description that distinguishes observed surface from unknown space, and iterative geometric rechecking, which re-encodes each candidate in the same representation so that discrepancies point to program edits while unobserved regions remain open to completion. Visual feedback and physical authoring guidance complete the modeling process, which produces articulated USD assets with explicit physical properties that load into Isaac Sim without manual adjustment. Experiments demonstrate leading articulation recovery on two benchmarks and validate USDCraft's effectiveness for real-to-sim-to-real robot manipulation.

Problem

• Articulated 3D Assets
• Real-to-Sim Robot Manipulation
• Mesh Reconstruction
• Programmatic Modeling

Innovation

• Programmatic Modeling
• Large Language Model
• Articulated 3D Assets
• Source Geometry Analysis
• Sim-to-Real Transfer