ST-WAM: Semantic-Temporal World Action Model for Robust Manipulation under Visual Distribution Shifts

2026-07-30

AI Summary

This work addresses the limited robustness and training-distribution hallucination of world models under visual distribution shifts, which stem from pixel-level generative supervision. To overcome these issues, the authors propose a semantic-temporal modeling framework that leverages DINOv3 to extract semantic features and combines them with a VAE to model fine-grained dynamics, establishing a shared semantic representation for both future prediction and historical retrieval. Key innovations include a Dual-Space Future Experts module that jointly predicts VAE latent variables and DINO features, and a Current-Anchored Intent Retrieval mechanism, enabling end-to-end training without additional pretraining or task-specific annotations. Experiments demonstrate state-of-the-art performance, achieving 98.7% and 92.8% success rates on LIBERO and RoboTwin 2.0, respectively, a 21.3-percentage-point improvement in zero-shot transfer to LIBERO-Plus, and a substantial increase from 25.8% to 61.5% success under real-world visual distribution shifts.

Abstract

World Action Models (WAMs) have emerged as a promising paradigm by jointly modeling robot actions and future visual dynamics. However, their reliance on pixel-generative future supervision can entangle action-relevant state transitions with task-irrelevant visual content, limiting robustness under visual distribution shifts. We identify Training-Distribution Hallucination, a recurring phenomenon in which futures conditioned on visually shifted observations hallucinate training-domain content rather than remain faithful to the current scene. A controlled frame-triplet diagnosis further shows that DINOv3 features remain more stable across visual shifts while better preserving task-state distinctions than Wan-VAE latents. Rather than correcting the predicted futures, we propose Semantic-Temporal WAM (ST-WAM) to improve action robustness by using DINOv3 as a shared semantic representation for future prediction and history retrieval while retaining fine-grained VAE dynamics. Its Dual-Space Future Experts (DSFE) jointly predict future VAE latents and DINO features, while Current-Anchored Intent Retrieval (CAIR) retrieves task-relevant evidence from recent DINO history under the current visual-language context. ST-WAM is trained end-to-end without additional embodied pretraining or task-specific annotations, and requires no explicit future generation at inference. It achieves 98.7% on LIBERO and 92.8% on RoboTwin 2.0; more importantly, compared with Fast-WAM, it improves zero-shot LIBERO-Plus performance by 21.3 percentage points and more than doubles real-world success under visual shifts from 25.8% to 61.5%. These results demonstrate that semantic-temporal modeling effectively complements pixel-generative dynamics for robust manipulation.

Problem

• visual distribution shifts
• World Action Models
• Training-Distribution Hallucination
• action robustness
• future prediction

Innovation

• Semantic-Temporal Modeling
• World Action Model
• Visual Distribution Shift
• DINOv3 Features
• Zero-shot Robustness