FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Rendering

2025-02-28

AI Summary

To address poor out-of-trajectory view reconstruction and rendering quality in autonomous driving—primarily due to the scarcity of high-quality out-of-trajectory supervision data—this paper introduces an Inverse View Warping (IVW) supervision mechanism and an online depth self-bootstrapping strategy. IVW leverages geometric backward mapping to generate compact, high-fidelity supervisory images, while the self-bootstrapping strategy jointly refines dense depth maps in real time using LiDAR and visual cues. Integrating 3D Gaussian splatting with end-to-end differentiable rendering, our method significantly improves both in- and out-of-trajectory rendering quality on the Waymo Open Dataset. On a custom-built simulation benchmark, it achieves a 2.1 dB PSNR gain over state-of-the-art methods for out-of-trajectory views—marking the first demonstration of high-fidelity out-of-trajectory reconstruction and effectively overcoming the sparsity limitation of LiDAR-derived depth.

Technology Category

• Computer Vision: 3D Computer Vision
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Driving scene reconstruction and rendering have advanced significantly using the 3D Gaussian Splatting. However, most prior research has focused on the rendering quality along a pre-recorded vehicle path and struggles to generalize to out-of-path viewpoints, which is caused by the lack of high-quality supervision in those out-of-path views. To address this issue, we introduce an Inverse View Warping technique to create compact and high-quality images as supervision for the reconstruction of the out-of-path views, enabling high-quality rendering results for those views. For accurate and robust inverse view warping, a depth bootstrap strategy is proposed to obtain on-the-fly dense depth maps during the optimization process, overcoming the sparsity and incompleteness of LiDAR depth data. Our method achieves superior in-path and out-of-path reconstruction and rendering performance on the widely used Waymo Open dataset. In addition, a simulator-based benchmark is proposed to obtain the out-of-path ground truth and quantitatively evaluate the performance of out-of-path rendering, where our method outperforms previous methods by a significant margin.

Problem

• Improves rendering quality for out-of-path viewpoints in driving scenes.
• Introduces Inverse View Warping for high-quality supervision in reconstruction.
• Proposes depth bootstrap strategy to enhance depth map accuracy.

Innovation

• Inverse View Warping for out-of-path view reconstruction
• Depth bootstrap strategy for dense depth maps
• Simulator-based benchmark for rendering evaluation