GeoSem-WAM: Geometry- and Semantic-Aware World Action Models

2026-06-02

AI Summary

Existing world models for action prediction often lack explicit modeling of environmental geometry and semantic context during inference, limiting their ability to comprehend complex scenes. This work proposes a structured world model that, for the first time, incorporates geometric and semantic predictions as auxiliary supervisory signals. By jointly learning future RGB, geometry, and semantics within a unified latent space, the model enhances its latent representations while refraining from explicitly generating future observations at test time to maintain computational efficiency. Through multitask learning and structured modeling, the approach significantly improves action prediction accuracy, scene understanding, and robustness in challenging environments, demonstrating the effectiveness of structured supervision for building efficient and scalable world models for action prediction.

Abstract

Recent World Action Models (WAMs) have demonstrated impressive capabilities in embodied decision-making. However, whether their effectiveness stems from explicit future imagination during inference or representation learning induced by predictive training remains an open question. Emerging evidence suggests the primary advantage lies in learning robust latent representations rather than generating future observations at test time. Nevertheless, existing WAMs mainly rely on RGB-based future prediction, which provides limited structural and spatial understanding of complex environments. To address this, we propose a structured world modeling framework that enhances latent representations through geometric and semantic supervision. Alongside future RGB prediction, our model introduces two auxiliary prediction branches for future geometry and semantic representations, enabling it to jointly capture scene dynamics, spatial geometry, and semantic context within a unified latent space. Crucially, our approach preserves efficient inference by avoiding explicit future rollout or video generation at test time. Extensive experiments show that incorporating structured world supervision consistently improves action prediction accuracy, scene understanding, and robustness under challenging embodied scenarios, highlighting its potential for advancing scalable and efficient WAMs.

Problem

• World Action Models
• structured world modeling
• spatial geometry
• semantic context
• embodied decision-making

Innovation

• World Action Models
• Geometric Supervision
• Semantic Supervision
• Latent Representation Learning
• Structured World Modeling