CARLA2Real: a tool for reducing the sim2real gap in CARLA simulator

2024-10-23

arXiv.org

AI Summary

To bridge the significant sim2real visual domain gap between CARLA and real-world driving scenes, this paper introduces CARLA2Real—the first real-time style transfer plugin natively integrated into CARLA’s rendering pipeline. Leveraging advanced image translation architectures—including GAN-based and diffusion-guided methods—it enables near-real-time (13 FPS) cross-domain stylization, dynamically aligning synthetic images to the visual distributions of real-world datasets such as Cityscapes, KITTI, and Mapillary Vistas. The open-source, plug-and-play plugin preserves semantic ground truth end-to-end and supports synchronized generation of augmented data and annotations. Experiments demonstrate that models trained on CARLA2Real-enhanced synthetic data achieve substantially improved generalization performance on real-world domains—particularly for feature extraction and semantic segmentation—thereby empirically validating its effectiveness in mitigating the sim2real gap.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Application Domains: Mobility, Driving & Flight

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Web data generation and simulation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Simulators are indispensable for research in autonomous systems such as self-driving cars, autonomous robots and drones. Despite significant progress in various simulation aspects, such as graphical realism, an evident gap persists between the virtual and real-world environments. Since the ultimate goal is to deploy the autonomous systems in the real world, closing the sim2real gap is of utmost importance. In this paper, we employ a state-of-the-art approach to enhance the photorealism of simulated data, aligning them with the visual characteristics of real-world datasets. Based on this, we developed CARLA2Real, an easy-to-use, publicly available tool (plug-in) for the widely used and open-source CARLA simulator. This tool enhances the output of CARLA in near real-time, achieving a frame rate of 13 FPS, translating it to the visual style and realism of real-world datasets such as Cityscapes, KITTI, and Mapillary Vistas. By employing the proposed tool, we generated synthetic datasets from both the simulator and the enhancement model outputs, including their corresponding ground truth annotations for tasks related to autonomous driving. Then, we performed a number of experiments to evaluate the impact of the proposed approach on feature extraction and semantic segmentation methods when trained on the enhanced synthetic data. The results demonstrate that the sim2real gap is significant and can indeed be reduced by the introduced approach.

Problem

• Reducing sim2real appearance gap in autonomous driving simulators
• Enhancing CARLA simulator output with real-world visual realism
• Improving synthetic data quality for autonomous system training

Innovation

• Enhances CARLA simulator's photorealism with real-world style
• Provides near real-time visual translation at 13 FPS
• Generates synthetic datasets with ground truth annotations