Dueling World Models: Advantage-Style Action Channels for Common-Mode Distractor Rejection

2026-08-06

AI Summary

In environments with uncontrollable dynamic disturbances, latent-variable world models often suffer from action blindness, failing to accurately capture the influence of actions on states. This work proposes a minimalist approach inspired by dueling value functions: during the readout phase of latent dynamics modeling, it subtracts the mean action effect to disentangle common-mode interference and preserve a clean, controllable action channel. The method requires no reconstruction, reward signals, or auxiliary losses and is compatible with any pretrained action-conditioned world model. Theoretical analysis demonstrates exact disturbance cancellation under finite samples, and experiments across grid worlds, synthetic data, continuous control, and Atari pixel environments confirm near-zero interference leakage and consistent improvements in goal-directed control performance.

Abstract

Latent world models plan by predicting future states from an action, but when a scene contains motion the agent does not control, they quietly go action-blind: predictions for different actions become indistinguishable even as the training loss keeps improving. Existing remedies suppress this distraction with reconstruction, task reward, or auxiliary objectives, each adding machinery or assumptions. We show that a minimal alternative suffices, borrowed from the dueling decomposition of value into a state baseline and an action advantage: in latent dynamics, subtracting a prediction's mean effect over actions cancels whatever the actions share--the action-independent variation where distractors live--leaving a clean, controllable channel, with no reward, no reconstruction, and no distractor-specific auxiliary loss. Because this is only a subtraction at readout time, it applies unchanged to any action-conditioned world model, including frozen pretrained ones. Across a gridworld, synthetic generators with known factors, distracting continuous control, and natural-pixel Atari, the isolated channel recovers the agent's own effect where entangled predictors fail, with nuisance leak indistinguishable from zero; applied post hoc it surfaces an action channel in off-the-shelf models that their raw readouts miss, and it converts into goal-reaching control in the gridworld. We prove the cancellation is exact in finite samples for both discrete and sampled action sets, and we state its measured boundary--distractors whose motion tracks the action--together with the remaining limitations in the appendix.

Problem

• world models
• distractor rejection
• action-conditioned prediction
• latent dynamics
• uncontrollable motion

Innovation

• Dueling World Models
• Action Advantage
• Distractor Rejection
• Latent Dynamics
• Action-Conditioned Prediction