unreal engine integration

To bridge the significant sim2real visual domain gap between CARLA and real-world driving scenes, this paper introduces CARLA2Real—the first real-time style transfer plugin natively integrated into CARLA’s rendering pipeline. Leveraging advanced image translation architectures—including GAN-based and diffusion-guided methods—it enables near-real-time (13 FPS) cross-domain stylization, dynamically aligning synthetic images to the visual distributions of real-world datasets such as Cityscapes, KITTI, and Mapillary Vistas. The open-source, plug-and-play plugin preserves semantic ground truth end-to-end and supports synchronized generation of augmented data and annotations. Experiments demonstrate that models trained on CARLA2Real-enhanced synthetic data achieve substantially improved generalization performance on real-world domains—particularly for feature extraction and semantic segmentation—thereby empirically validating its effectiveness in mitigating the sim2real gap.

Existing autonomous driving simulators face two key limitations: insufficient scenario diversity in graphics-based engines (e.g., CARLA) and poor generalizability in learning-based methods (e.g., NeuSim), which are restricted to specific object categories and require dense multi-sensor annotations. To address these bottlenecks, we propose a real2sim2real end-to-end scalable simulation framework. Our method integrates 3D generative modeling, real-to-sim domain translation, forward multi-sensor simulation, and inverse rendering to establish a closed loop: automatically mining rare driving scenarios from real-world data, generating high-fidelity, category-agnostic 3D object assets, and synthesizing corresponding multi-modal sensor data. Crucially, it operates without category priors or dense annotations, significantly improving rare-scenario coverage and data efficiency. Experiments demonstrate that the synthesized data substantially outperforms both conventional computer graphics–based and learning-based baselines in training perception models for robustness.

This work addresses the inefficiency in autonomous driving testing caused by cumbersome workflows and redundant code during complex scenario construction. To overcome these limitations, the authors propose Modular2Simple, a novel tool that introduces a modular-composition paradigm for scenario generation. By combining simple or modular OpenSCENARIO scenes, Modular2Simple enables flexible and efficient creation of diverse, complex test scenarios while strictly adhering to the OpenSCENARIO standard. The approach seamlessly integrates with mainstream simulation platforms such as CARLA, significantly enhancing scenario reusability and customizability while reducing development complexity. Experimental results demonstrate that, compared to conventional methods, the proposed solution substantially decreases both development time and labor costs, markedly improving the efficiency and diversity of test scenario construction.

Evaluating autonomous driving systems requires realistic, diverse, and efficiently generated safety-critical edge-case scenarios (e.g., occluded pedestrians crossing, sudden cut-ins), yet existing approaches struggle to simultaneously satisfy fidelity, variability, and scalability. Method: This paper proposes an end-to-end driving scenario video generation framework powered by large language models (LLMs). It employs few-shot prompting to elicit structured safety-event scripts from LLMs; integrates CARLA for high-fidelity physics simulation and multi-agent coordination; and leverages Cosmos-Transfer1 and ControlNet to build a controllable rendering pipeline that transforms synthetic images into photorealistic videos. Contribution/Results: Experiments demonstrate that the framework automatically generates diverse, high-fidelity video sequences covering rare hazardous scenarios. It significantly improves simulation-to-reality transfer capability and enhances the robustness evaluation efficacy of autonomous driving systems, enabling more rigorous and scalable safety validation.

Existing driving world models prioritize generative fidelity and controllability but overlook their practical utility for downstream perception tasks—particularly extreme-scenario detection. This paper introduces Dream4Drive, the first driving world model explicitly designed as a perception-oriented synthetic data generator. It employs 3D perception-guided graph modeling and multi-view rendering to produce high-fidelity, editable RGB and multimodal video sequences. We further release DriveObj3D, the first large-scale 3D driving asset dataset. Experiments demonstrate that perception models trained solely on Dream4Drive-synthesized data consistently outperform real-data baselines—under both identical and double the real-data training epochs—with particularly pronounced gains in extreme-case recognition. This work establishes a rigorous validation paradigm for assessing the efficacy of synthetic data in autonomous driving perception.

This work addresses the challenges of automatically generating complete, executable 3D games within commercial game engines—namely, procedural complexity and high technical barriers—by proposing AutoUE, a multi-agent system that coordinates multiple specialized agents to end-to-end produce functional 3D games, encompassing scene construction, gameplay logic, and interactive code synthesis. The approach innovatively integrates retrieval-augmented generation to mitigate tool hallucinations in large language models, while incorporating constraints from Unreal Engine documentation and established game design patterns to ensure code correctness. An automated testing mechanism is further introduced to validate dynamic game behaviors. Evaluated on a newly curated dataset for game generation, experiments demonstrate AutoUE’s effectiveness in producing fully functional 3D games and confirm the overall system performance.

This work addresses the critical need for generating extreme driving scenarios that simultaneously exhibit visual photorealism, semantic plausibility, and physical feasibility—qualities rarely achieved together by existing methods. The authors propose a modular synthesis framework that decouples high-level semantic control from low-level physical execution. Specifically, they construct editable 3D Gaussian scenes from real-world data, employ a multi-agent large language model to generate high-risk interactive intent trajectories, and execute these trajectories under dynamic constraints using a PID controller in CARLA. The resulting motions are back-projected into the Gaussian scene to render first-person-view videos. This framework is the first to unify semantic controllability, physical realism, and visual fidelity within a single system, demonstrating strong spatiotemporal consistency, semantic alignment, and photorealism on the Waymo Open Dataset.

Existing high-fidelity simulators struggle to meet the demands of embodied intelligence research in terms of generality, programmability, and rendering efficiency. To address this gap, this work proposes SPEAR—a Python-based simulation library built on Unreal Engine—that exposes over 14,000 UE functions to Python for the first time, enabling unprecedented procedural control. SPEAR employs a modular plugin architecture and a high-level task graph scheduling model to support deterministic execution of complex tasks, delivering high-definition multimodal images along with accurate annotations to NumPy arrays at an efficient rate of 73 FPS. The system has been successfully applied to diverse tasks including agent control, city-scale rendering, multi-view facial imaging, physics-informed co-simulation, and language-driven scene editing, significantly enhancing both simulation efficiency and functional coverage.

Existing autonomous driving simulation methods struggle to balance photorealism and controllability in scene generation. This work proposes a Gaussian-grid hybrid representation to construct a unified, editable 3D scene framework that supports language-driven vehicle insertion and fine-grained trajectory manipulation. To ensure spatiotemporal consistency, the method incorporates noise-aware video diffusion rendering and operates under a single-pass feedforward editing paradigm—eliminating the need for per-trajectory optimization. To the best of our knowledge, this is the first approach to achieve high-fidelity, arbitrarily controllable driving scene editing. Experiments demonstrate an 83.4% improvement in user preference and a 25.19-point reduction in FID over the next-best method. The authors also introduce HorizonSuite, a comprehensive benchmark for evaluating controllable driving simulation.

Existing methods for generating 3D vehicle models rely on synthetic data, suffering from domain shift, arbitrary poses, and undefined scale, which lead to poor visual consistency in real-world driving scenarios. This work proposes a two-stage self-supervised framework that first trains a reconstruction network using pose-annotated images and then enables 3D reconstruction from unposed real-world images by predicting camera parameters. The approach incorporates scale-awareness and appearance harmonization modules to ensure geometric and photometric fidelity. To the best of our knowledge, this is the first method to generate simulation-ready 3D vehicle assets—exhibiting consistent poses, physically plausible scale, and lighting-coherent appearance—using only unposed real driving images in an end-to-end manner, thereby significantly enhancing the quality and scalability of autonomous driving simulation assets.