Real-to-Sim Generation: Synthesizing Scenario Programs from Real-World Data via Constraint Solving

2026-09-20

AI Summary

为解决仿真与现实差距问题，提出R2SGEN框架，通过解约束方法从真实数据合成场景程序，加速搜索过程并保证一致性。

Abstract

The demand for synthetic training data is hindered by the sim-to-real gap, as current data-driven and LLM-based generators often produce physically implausible scenarios. To address this, we propose R2SGEN, a Real-to-Sim framework that synthesizes structured scenario programs from real-world data. To overcome the combinatorial explosion and intractability of monolithic Satisfiability Modulo Theories (SMT) encoding, we introduce a decoupled synthesis strategy. This approach separates the discrete structural program search from continuous geometric resolution using lightweight, atomic SMT constraints. Furthermore, we significantly accelerate the search process by integrating two tailored pruning mechanisms: Common Prefix Abstraction-based pruning for Breadth-First Search and Branch-and-Bound for Depth-First Search. We evaluate R2SGEN on 20 real-world scenes of varying complexity from the nuScenes dataset. Experimental results show that our method guarantees consistency with the input scene and produces substantially lower-cost programs than the LLM-based baselines under the evaluated inputs. Both proposed search paradigms exhibit complementary advantages, proving highly efficient and scalable for high-complexity synthetic data generation.

Problem

• sim-to-real gap
• synthetic training data
• physically implausible scenarios

Innovation

• R2SGEN
• Decoupled Synthesis Strategy
• SMT Constraints
• Pruning Mechanisms
• Common Prefix Abstraction