What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations

2026-07-29

AI Summary

This work investigates whether latent-variable world models can learn physical parameters in predictive tasks and elucidates the mechanisms underlying their identifiability. By conducting controlled interventions in POKEWORLD and RH20T environments, the study introduces a certificate-gating protocol that integrates multimodal perception, deterministic prediction objectives, supervised readout heads, and recoverability certification to map the identifiability of physical parameters. The findings reveal that the choice of input modalities and prediction targets jointly determines whether latent representations encode physical knowledge: stiffness is effectively captured only when tactile signals are predicted (R²=0.50), visual one-step prediction overlooks visible states, and drag force remains poorly learned across all settings (R²≈0.13). Full multimodal prediction substantially outperforms persistence baselines on RH20T, with performance scaling favorably with dataset size.

Abstract

A central premise of latent world models is that predicting the future forces a representation to internalize the physics of its environment. Which physical quantities does a trained latent actually contain, and what decides this? We answer with controlled interventions in POKEWORLD, an interactive environment whose visually identical objects hide mass, drag, and contact stiffness. A certificate-gated protocol first certifies each parameter as recoverable from raw observations, then measures whether it enters the latent, so a null result can be attributed to the objective rather than to the environment. The resulting identifiability map has two organizing mechanisms and one frontier. Inputs limit what can be known, while prediction targets decide what is retained. Stiffness enters the latent only when touch is forecast ($R^2=0.50$, compared with $-0.02$ when the same signal is merely fused into the input), and under single-step prediction a vision-only latent discards even perfectly visible object state. Drag marks the frontier. It carries a recoverability certificate of 0.89 yet plateaus near 0.13 under every deterministic prediction objective we test, while a supervised head on the same trunk reaches 0.45. Parameters whose readout is slow and ratio-type under the sensed coordinates fall outside what these objectives acquire. On RH20T, an input-target factorial across scaling curves reproduces both mechanisms across two robots and 4,258 episodes. Every arm missing information or prediction pressure stays flat over a fivefold data range, and only the full multimodal objective forecasts force beyond a persistence baseline, with held-out gains that grow with scale. Objective structure determines which physical parameters a latent acquires, and additional data improves only the parameters it already acquires.

Problem

• latent world models
• physical parameter identifiability
• multimodal predictive representations
• parameter recoverability
• prediction objectives

Innovation

• latent world models
• physical parameter identifiability
• multimodal prediction
• certificate-gated protocol
• predictive representation