DeltaWorld: Physically Consistent Interactive World Simulators via Action-Conditioned Latent Increment Learning

2026-10-01

AI Summary

This study addresses the difficulty of existing world models in capturing subtle action-induced state changes, which often leads to physical inconsistencies. We propose a novel modeling paradigm that predicts latent feature deltas rather than directly generating next states, enabling the construction of physically consistent interactive world simulators. The core methodology incorporates a Delta-LTM incremental learning mechanism, interaction-aware latent alignment, and counterfactual interaction region supervision, effectively suppressing artifacts such as object penetration and excessive deformation. Experimental evaluations demonstrate that our approach reduces FVD by 46.6% and LPIPS by 31.1% across cross-robot datasets, significantly improving both the physical consistency and visual quality of long-horizon predictions.

Abstract

Interactive world simulators can provide scalable environments for robot planning, policy training, and evaluation by predicting action consequences while reducing reliance on repeated physical rollouts. To serve these applications, they must generate future image sequences that respond faithfully to robot actions and preserve the dynamics of robot-object interactions over long horizons. However, existing world models typically predict the entire next latent state and often fail to capture subtle changes induced by robot actions. Such omissions can produce physically implausible outcomes, including object interpenetration and excessive deformation. To address this limitation, we propose DeltaWorld, a physically consistent interactive world simulator for robotic manipulation. Our method introduces the Delta Latent Transition Model (Delta-LTM), which predicts action-induced latent feature changes and adds them to the current latent state to obtain the next state, rather than predicting the next latent state directly. To mitigate object interpenetration and excessive deformation in predicted future frames, Interaction-aware Latent Alignment is introduced to construct counterfactual interaction regions and supervise interaction-related latent changes. DeltaWorld is evaluated on the IWS manipulation benchmark and a self-collected cross-robot dataset covering multiple robot embodiments and manipulation tasks. On the cross-robot dataset, DeltaWorld reduces FVD by 46.6% and LPIPS by 31.1% relative to the IWS baseline. These results highlight the potential of DeltaWorld for long-horizon action-conditioned video prediction in robotic manipulation.

Problem

• world simulator
• robotic manipulation
• physical consistency
• latent state prediction
• long-horizon video prediction

Innovation

• Action-Conditioned Latent Increment Learning
• Delta Latent Transition Model
• Interaction-aware Latent Alignment
• Interactive World Simulators
• Robotic Manipulation