SIMSplat: Predictive Driving Scene Editing with Language-aligned 4D Gaussian Splatting

2025-10-02

AI Summary

Existing driving scene editing methods struggle to efficiently generate photorealistic, dynamically interactive virtual environments. This paper introduces the first language-guided driving scene editing framework, integrating 4D Gaussian splatting modeling, multi-agent motion prediction, and differentiable rendering to enable fine-grained, natural-language-instruction-driven scene manipulation—including object insertion/deletion, trajectory editing, and predictive path optimization. The core innovation lies in a language–3D scene alignment mechanism that ensures semantic consistency and spatiotemporal dynamic coherence. Experiments on the Waymo Open Dataset demonstrate that the system efficiently generates diverse, semantically plausible, and interactionally realistic complex driving scenes. It significantly improves both editing accuracy and generation efficiency compared to prior approaches.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Driving scene manipulation with sensor data is emerging as a promising alternative to traditional virtual driving simulators. However, existing frameworks struggle to generate realistic scenarios efficiently due to limited editing capabilities. To address these challenges, we present SIMSplat, a predictive driving scene editor with language-aligned Gaussian splatting. As a language-controlled editor, SIMSplat enables intuitive manipulation using natural language prompts. By aligning language with Gaussian-reconstructed scenes, it further supports direct querying of road objects, allowing precise and flexible editing. Our method provides detailed object-level editing, including adding new objects and modifying the trajectories of both vehicles and pedestrians, while also incorporating predictive path refinement through multi-agent motion prediction to generate realistic interactions among all agents in the scene. Experiments on the Waymo dataset demonstrate SIMSplat's extensive editing capabilities and adaptability across a wide range of scenarios. Project page: https://sungyeonparkk.github.io/simsplat/

Problem

• Enables intuitive driving scene editing using natural language prompts
• Supports detailed object-level manipulation and trajectory modifications
• Incorporates predictive path refinement for realistic agent interactions

Innovation

• Language-controlled editing with Gaussian splatting
• Direct object querying in reconstructed driving scenes
• Predictive path refinement using multi-agent motion prediction