R3ST: A Synthetic 3D Dataset With Realistic Trajectories

2025-12-18

AI Summary

Existing synthetic datasets lack realistic vehicle motion trajectories, while real-world datasets struggle to provide precise, scalable 3D annotations. Method: We introduce the first synthetic 3D traffic dataset integrated with authentic driving trajectories—leveraging high-fidelity vehicle trajectories captured by drone-based SinD data to construct programmable virtual road environments in CARLA/Unity, enabling seamless coupling of real-world motion patterns with synthetic scenes. Our framework supports fully automatic, multi-modal ground truth generation—including 3D bounding boxes, velocity, and driver intent—as well as multi-view rendering. Contribution/Results: Compared to purely synthetic or purely real datasets, ours preserves human-like driving behavior while significantly improving cross-domain generalization and behavioral plausibility of trajectory prediction models. Extensive evaluation across multiple benchmarks demonstrates its complementary advantages, effectively bridging the long-standing gap between synthetic data and realistic motion modeling.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Web data generation and simulation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Datasets are essential to train and evaluate computer vision models used for traffic analysis and to enhance road safety. Existing real datasets fit real-world scenarios, capturing authentic road object behaviors, however, they typically lack precise ground-truth annotations. In contrast, synthetic datasets play a crucial role, allowing for the annotation of a large number of frames without additional costs or extra time. However, a general drawback of synthetic datasets is the lack of realistic vehicle motion, since trajectories are generated using AI models or rule-based systems. In this work, we introduce R3ST (Realistic 3D Synthetic Trajectories), a synthetic dataset that overcomes this limitation by generating a synthetic 3D environment and integrating real-world trajectories derived from SinD, a bird's-eye-view dataset recorded from drone footage. The proposed dataset closes the gap between synthetic data and realistic trajectories, advancing the research in trajectory forecasting of road vehicles, offering both accurate multimodal ground-truth annotations and authentic human-driven vehicle trajectories.

Problem

• Generates synthetic 3D environment with real-world vehicle trajectories
• Provides accurate multimodal ground-truth annotations for traffic analysis
• Advances trajectory forecasting research by combining synthetic data realism

Innovation

• Generates synthetic 3D environment with realistic trajectories
• Integrates real-world trajectories from drone footage dataset
• Provides accurate multimodal ground-truth annotations