VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving

2026-10-07

AI Summary

This study addresses the limitation of existing world models in effectively conditioning on driving intent and actions, which hinders their ability to predict alternative future scenarios under varying maneuvers. To this end, we propose VGGTWorld-VLA, a framework that achieves controllable 3D world evolution prediction by injecting driving semantics and ego-vehicle motion representations. Specifically, the method introduces an action-semantics joint mechanism and a geometry-language-action bridging module, integrating 3D geometric reconstruction, Transformer-based temporal modeling, and multimodal features. This enables the model to generate distinct future geometric predictions from identical observations conditioned on alternative actions. Experimental results demonstrate that the proposed approach achieves competitive performance on the NAVSIM benchmark, while ablation studies validate the effectiveness of semantic and action conditioning for controllable world prediction.

Abstract

VGGT provides a strong foundation for geometry-centric world models by recovering unified 3D scene geometry from visual observations. Although recent extensions enable temporal 3D prediction, their future evolution remains weakly conditioned on driving intentions and actions, limiting their ability to model alternative action-dependent futures. We propose VGGTWorld-VLA, an intention-conditioned extension of VGGT-World for controllable 3D world evolution in autonomous driving. First, we introduce an action--semantic conditioning mechanism that injects complementary driving semantics and ego-motion representations into the future-token stream, enabling different future geometry predictions for the same observed scene under alternative ego actions. Second, we develop a geometry--language--action bridge that adapts historical geometry, VLA semantic features, and maneuver and trajectory representations for joint conditioning of future geometry prediction. We evaluate future geometry prediction on NAVSIM, while conditioning ablations further examine the contributions of semantic and action information. Compared with the baseline, our method demonstrates competitive geometry prediction performance. Ablation studies further support the effectiveness of semantic and action conditioning. These results demonstrate the potential of semantic and action conditioning for controllable VGGT-based world prediction in autonomous driving.

Problem

• Autonomous Driving
• 3D World Model
• Intent Conditioning
• Future Prediction
• Controllable Generation

Innovation

• World Model
• Action-Semantic Conditioning
• 3D Geometry Prediction
• Vision-Language-Action
• Autonomous Driving