CoDimRecon: Agentic Reconstruction of Sim-Ready 3D Scenes with Deformable Curves, Surfaces, and Volumes

2026-09-28

AI Summary

This work addresses the challenge of reconstructing simulation-ready 3D scenes containing deformable objects from multi-view images by proposing a novel agent-based framework. Taking multi-view RGB inputs and scene-level geometric priors, the framework introduces a dimension-based reconstruction strategy for deformable objects. It integrates behavioral testing to automatically refine physical models and geometric details, enabling object-level mesh generation for rigid, articulated, and deformable entities. The system outputs simulation-compatible curve, surface, and volumetric geometries. Evaluated on datasets such as Replica, it achieves high-fidelity reconstruction and successfully supports rod, shell, and solid simulations alongside robotic interaction demonstrations. Ultimately, this project establishes a new paradigm for constructing editable simulation environments.

Abstract

Reconstructing simulation-ready 3D scenes from real-world observations enables robotics, gaming, and immersive applications, yet existing methods largely assume rigid objects. This leaves an important gap for deformables, whose simulation-ready geometry depends on dimensionality (curves, surfaces, or volumes) and whose behavior may require models beyond elasticity. We present CoDimRecon, an agentic framework that reconstructs editable scenes containing rigid, articulated, and deformable objects from multi-view RGB observations. Scene-level geometric priors ground scale and layout, while object-level generated meshes guide the agent toward detailed, compact geometry; articulated rigid objects are decomposed into movable parts with explicit joints. For deformables, category-wise agent sessions reconstruct curves as centerlines with radii, surfaces as manifold shells with thickness, and volumes as watertight solids for volumetric meshing. Reusable simulator skills initialize compatible physical models and parameters, while agent-guided behavioral tests expose mismatches and trigger targeted revisions of motion, geometry, numerics, or material modeling. On evaluated Replica and ScanNet++ scenes, CoDimRecon achieves competitive compositional reconstruction accuracy while additionally producing deformable assets for rod, shell, and solid simulation. We further demonstrate robot interactions across all three representations, including a controlled paper-folding case in which behavioral testing motivates plastic bending.

Problem

• simulation-ready 3D reconstruction
• deformable objects
• multi-view RGB observations
• physical simulation
• co-dimensional geometry

Innovation

• Agentic Framework
• Deformable Reconstruction
• Simulation-Ready 3D Scenes
• Cross-Dimensional Representation
• Behavioral Testing