Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving

2026-07-31

AI Summary

This work addresses the limitation of conventional autonomous driving world models that reconstruct full future scenes while overlooking elements critical to decision-making. The authors propose Auto-JEPA, an action-oriented implicit world model that leverages a joint-embedding predictive architecture to directly learn driving intent embeddings aligned with future trajectories from visual observations, ego-vehicle history, and navigation commands. Instead of dense future scene modeling, Auto-JEPA retrieves executable trajectories from a fixed trajectory memory bank, eliminating the need for explicit perception annotations or a learnable trajectory generator. The visual encoder is frozen to enhance computational efficiency and generalization. Evaluated on NAVSIM v1 and v2, the method achieves 91.3 PDMS and 89.1 EPDMS, respectively. Semantic occlusion experiments further demonstrate its ability to effectively attend to key dynamic objects influencing driving decisions.

Abstract

Existing autonomous-driving world models typically perform dense prediction of future videos, occupancy states, BEV representations, or agent motion. We argue that planning need not reconstruct the complete future world, but only focus on scene features that affect future ego action. Based on this perspective, we propose Auto-JEPA, an action-oriented latent world model that learns continuous future driving intent through joint-embedding prediction. Given visual observations, egomotion history, and navigation commands, Auto-JEPA predicts an intent embedding aligned with the latent representation of the future ego trajectory. The predicted intent retrieves executable trajectories from a fixed trajectory memory, which are then ranked by a scene-conditioned candidate selection module. Auto-JEPA keeps the visual encoder frozen, requires no explicit perception annotations, and uses no learned trajectory generator. By optimizing only task-specific modules for trajectory representation, intent prediction, and candidate selection, Auto-JEPA achieves 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2. Semantic occlusion experiments show that masking dynamic-agent regions induces an average intent change 2.97x that of equal-area random masking. Moreover, occluding vehicles that affect future driving substantially changes the predicted intent and selected trajectory, whereas both remain essentially unchanged when non-influential vehicles are occluded. These results show that future-intent prediction encourages the model to focus on planning-relevant visual features and supports high-quality planning without dense future-world modeling.

Problem

• autonomous driving
• world model
• future intent
• planning
• latent representation

Innovation

• latent world model
• continuous intent
• joint-embedding prediction
• end-to-end autonomous driving
• trajectory memory