The Invisible Hand of Physics: When Video Diffusion Models Know More Than They Show

2026-06-03

AI Summary

This study investigates whether video diffusion models implicitly encode physical structure rather than merely reproducing motion patterns observed in training data. Through deterministic sampling inversion and latent trajectory reconstruction, combined with linear probing and attention analysis, the authors demonstrate that signals of physical plausibility emerge spontaneously within the denoising Transformer—not from the VAE input—and do so without any explicit self-supervised objective. The proposed approach achieves an average classification accuracy of 81.27% on the IntPhys and InfLevel benchmarks, substantially outperforming specialized representation learning baselines such as V-JEPA and VideoMAE.

Abstract

Modern video diffusion models generate increasingly realistic and temporally coherent videos, motivating their use as candidate world simulators. Yet it remains unclear whether these models internally encode physical structure, or merely reproduce motion patterns seen during training. We study this question by probing video diffusion models along latent trajectories corresponding to real videos with known physical plausibility. To obtain such trajectories, we approximately invert the deterministic sampling process by integrating the learned velocity field backward from a clean video latent to noise, giving access to the model's intermediate states and attention maps. Using these recovered trajectories, we show that physical plausibility is linearly decodable from diffusion transformer states across IntPhys and InfLevel, reaching around 81.27% average accuracy and outperforming dedicated representation-learning baselines such as V-JEPA and VideoMAE. Surprisingly, this signal is absent from the VAE latent input and emerges inside the denoising transformer itself, despite the model not being trained with a self-supervised predictive objective. These findings suggest that physically meaningful representations can arise as a byproduct of generative denoising.

Problem

• video diffusion models
• physical plausibility
• latent trajectories
• world simulation
• representation learning

Innovation

• video diffusion models
• physical plausibility
• latent trajectory inversion
• diffusion transformer
• emergent representation