Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

2026-07-21

AI Summary

This work addresses the limitations of existing Real2Sim approaches, which rely heavily on manual intervention and struggle to efficiently construct high-fidelity physics-based simulation environments. The authors propose the first unified framework that leverages a vision–language agent to automatically translate real-world videos of robot–object interactions into fully simulatable scenes, supporting diverse interaction types including rigid bodies, deformable objects, and human-like motions. By integrating geometric reconstruction, physical parameter inference, state reasoning, and an open-source vision–language model, the system achieves end-to-end automation. Experiments demonstrate that the method successfully reproduces a wide range of complex interactive scenarios with high fidelity, significantly reducing reliance on large-scale models and providing a high-quality, low-cost simulation foundation for robot policy learning and evaluation.

Abstract

Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on manual tuning of visual foundation models, mesh cleanup, coordinate-frame alignment, and brittle workflow glue across visual perception tools and simulators. We introduce \textit{Agentic Real2Sim}, a framework for generalized physical world modeling with vision-language agents, converting a real-world recording of object-robot interaction into a simulatable episodic twin which preserves observations, geometries, robot interactions, and object states. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines, marking a first step toward scalable conversion. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining comparable conversion success rate. We aim to use the resulting real-world-aligned twins for downstream robotics tasks, specifically policy learning and evaluation. The project site is available at https://ericchen321.github.io/agentic_real2sim.github.io/.

Problem

• Real2Sim
• physics-based world modeling
• robotic interaction
• simulation conversion
• object state recovery

Innovation

• Agentic Real2Sim
• vision-language agents
• physics-based simulation
• real-to-sim conversion
• robotic world modeling