Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise

2026-09-28

AI Summary

This study addresses the labor-intensive nature of designing assets, layouts, and parameters for physics simulation scenes, which hinders the rapid generation of executable dynamic scenarios from text. To overcome this, we propose a hierarchical agent pipeline built upon the Genesis engine, featuring a Planner-Writer-Critic architecture that constitutes the first agent framework specifically tailored for simulation. Furthermore, we introduce compact Debug Cards distilled from graphical demonstrations to enable character-specific physical guidance and automated execution repair. Across 42 evaluation tasks, our method consistently surpasses existing baselines in both physical fidelity and visual quality. User preference studies under blind testing conditions demonstrate significant improvements, while the framework effectively supports multimodal downstream applications such as dataset construction.

Abstract

Creating diverse physical simulations remains labor-intensive because assets, layout, physical parameters, motion, control, and rendering must be designed and debugged jointly. We present Text2Sim, a simulation-specialized agentic pipeline that converts a text-only request into an executable, editable dynamic case. Built on Genesis, Text2Sim uses a hierarchical agentic structure that combines a Planner with specialized Writers, asset-generation tools, and an independent Critic. Compact skills (Debug Cards) distilled from graphics demonstrations provide role-specific physical guidance for execution-based repair. We evaluate physical quality, visual quality, and human preference on 42 held-out prompts spanning rigid, articulated, deformable, and cloth phenomena, with a paper-level split between experience construction and evaluation. We design automatic physical and visual scorers to evaluate the quality of the results, and Text2Sim achieves higher scores than all four state-of-the-art baselines on both metrics. In blinded user studies with these baselines, significantly more participants prefer Text2Sim than prefer the baselines, which is consistent with the results from our automatic scorers. The pipeline also supports a broad range of downstream applications; we select dataset construction and extension to multimodal input as two representative examples. We will release the code, the Debug Card library, and a dataset of generated cases, each pairing the text prompt and rendered video with the executable program, assets, physical parameters, controls, and recorded states.

Problem

• physics-based simulation
• text-to-simulation
• simulation generation
• agentic pipeline

Innovation

• Agentic Pipeline
• Physics-Based Simulation
• Hierarchical Agents
• Knowledge Distillation
• Debug Cards