Modeling Latent Disturbances for Robust Decision-Making in World Models

2026-10-05

AI Summary

This study addresses the challenge of robust decision-making arising from the difficulty of quantifying latent-space uncertainty in world models. We propose a conformal prediction-based latent perturbation modeling approach that constructs trustworthy latent dynamic sets within a game-theoretic optimization framework. By integrating dynamics-aware similarity metrics with out-of-distribution detection, the method effectively balances pessimism and plausibility in decision-making. Both simulation and hardware experiments demonstrate that our approach reduces safety filter failure rates by 70% and sampling strategy overhead by 54%, achieving data-efficient robust decision-making and policy guidance in latent spaces.

Abstract

In this paper, we study robust decision-making in the latent space of world models (WMs). Robust optimization is a mathematical framework where, given explicitly specified dynamics and physically meaningful disturbances, a robot can select actions that remain effective even under worst-case disturbances. However, applying this principle to the learned latent space of WMs introduces a fundamental challenge: because WMs have fully learned state spaces and dynamics inferred from high-dimensional observations, it is unclear how to define latent-space disturbances that faithfully represent uncertainty in the underlying system. Our key idea is to model a latent-space disturbance as a perturbation to the learned latent dynamics that induces pessimistic but plausible transitions. Specifically, we construct a set of plausible latent dynamics by combining a dynamics-aware similarity metric that captures plausible transitions with out-of-distribution detection that excludes implausible latent states. We calibrate this uncertainty set over latent dynamics using conformal prediction, ensuring that WM imaginations induced by the latent disturbance remain plausible without becoming overly pessimistic. We then jointly optimize robust robot actions and the worst-case latent disturbances through game-theoretic optimization. We leverage this latent-space robust optimization to robustify policy steering, considering two paradigms: latent safety filtering and sample-and-verify steering of a generative control policy. Our controlled simulation experiments show that our latent disturbance enables robust decision-making directly in WM latent spaces, and hardware experiments with a Franka manipulator show that modeling latent disturbances enables robust policy steering, reducing failures by 70% in safety filtering and 54% in sampling-based policy steering. Project website: https://junwon.me/LatentDisturbance/.

Problem

• World Models
• Robust Decision-Making
• Latent Disturbances
• Uncertainty
• Latent Space

Innovation

• World Models
• Latent Disturbance
• Robust Decision-Making
• Conformal Prediction
• Game-Theoretic Optimization