RECAST: From Log Replay to Closed-Loop Driving Simulation with View-Complete Actors

2026-09-25

AI Summary

This study addresses the challenge of viewpoint-induced rendering artifacts and unreliable observations for dynamic actors in closed-loop driving simulation. To this end, it proposes a two-stage model adaptation strategy based on 3D Gaussian Splatting. By leveraging image-to-3D priors, the method generates view-consistent complete actors from single frames and registers them into the scene, thereby enabling planner-in-the-loop closed-loop interactive simulation. Furthermore, the RECAR dataset is constructed to facilitate controllable evaluation beyond logged trajectories. Experimental results demonstrate that the proposed approach significantly reduces the Fréchet distance while improving the CLIP score. In closed-loop evaluations, the method achieves a collision-free rate of 63% and increases the average Time-to-Collision (TTC) to 2.15 seconds, indicating substantially enhanced safety and realism in autonomous driving simulation.

Abstract

Closed-loop driving simulation requires rendered observations to remain reliable as the ego vehicle and surrounding actors move beyond their recorded trajectories, exposing views absent from the source log. Existing data-driven simulators reconstruct dynamic actors from sparse observations, which can result in rendering artifacts under these viewpoint changes. We introduce RECAST (REconstructing Controllable Actors for Simulation and Testing), a 3D Gaussian Splatting framework that generates a view-complete actor from a single segmented vehicle observation in a driving log and registers the generated actor in the reconstructed scene. RECAST supports planner-in-the-loop rendering under controlled ego-actor interactions. To adapt an image-to-3D prior to real vehicles, we further introduce RECAR, a dataset of approximately 20K real vehicles with 600K background-free RGBA images spanning diverse vehicle colors and types. We use two-stage adaptation to improve vehicle generation from real driving-log observations. At the actor level, RECAST reduces $\mathrm{FD}_{\mathrm{incep}}$ from 9.788 to 7.992 relative to unadapted TRELLIS. At the scene level, under actor motion beyond logged trajectories, RECAST reduces $\mathrm{FD}_{\mathrm{incep}}$ from 129.35 to 112.10 and increases $\mathrm{CLIP}_{\mathrm{margin}}$ ($\times1000$) from 0.14 to 3.47 relative to Street Gaussians. We demonstrate planner-in-the-loop simulation with the image-conditioned planner GTRS-Dense. Compared with native Street Gaussians actors, RECAST increases the no-collision (NC) rate from 22.2% (12/54) to 63.0% (34/54) and the mean minimum predicted time-to-collision (TTC) from 0.798 s to 2.150 s. These experiments show that RECAST supports closed-loop planner evaluation under controlled ego-actor interactions beyond log replay. Visit our project page at https://zijunkr.github.io/RECAST/

Problem

• closed-loop driving simulation
• view synthesis
• rendering artifacts
• dynamic actors
• log replay

Innovation

• Closed-Loop Driving Simulation
• 3D Gaussian Splatting
• View-Complete Actor Generation
• Domain Adaptation
• Planner-in-the-Loop