Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment

2026-06-03

AI Summary

While existing video generation models achieve high visual fidelity, they often violate physical laws. This work proposes PILA, a framework that injects physics-aware structure into frozen, pretrained flow-matching video generators by introducing physically structured latent variables. Specifically, an anchoring field estimator maps latents to proxy slots organized by physical attributes, and a mixture-of-experts mechanism enables fine-grained modeling. The approach innovatively incorporates label-prior masked expert routing and operational residual regularization to effectively handle the heterogeneity of real-world dynamics. Notably, PILA requires no backbone fine-tuning yet consistently achieves state-of-the-art performance in both visual quality and physical plausibility on VBench-2.0, VideoPhy-2, and PhyGenBench. Furthermore, adapters trained on small-scale models transfer directly to 14B-parameter large models, demonstrating strong scalability.

Abstract

Large-scale video generation models have made remarkable progress in semantic consistency and visual quality, producing videos that are increasingly coherent and visually convincing. Nevertheless, the dynamics induced by pixel-level fitting do not naturally accommodate the regularities that govern real-world motion and interaction, resulting in persistent shortcomings in physical plausibility. To address this limitation, we propose \textbf{PILA} (Physics-Informed Latent Alignment), a framework that injects physics-structured latent guidance into the frozen flow-matching dynamics of pretrained video models. Specifically, PILA first employs anchored field estimation to map frozen-generator latents into an operational physical attribute bank organized by field-proxy slots, using observable motion as a kinematic anchor for constructing less directly observed proxies. To handle the heterogeneity of real-world dynamics, PILA adopts a mixture-of-experts design over physical categories. Label-prior masked expert routing selects category-specific operator experts, whose refinements are regularized by operational residuals abstracted from physical relations. Finally, the refined proxies are fused into the physical attribute bank and decoded into a correction to the flow-matching vector field, injecting physics-aware guidance while preserving the visual prior of the pretrained backbone. With staged adapter training on Wan 2.1-1.3B and direct transfer of the learned adapter to Wan 2.2-14B, PILA achieves state-of-the-art results on VBench-2.0, VideoPhy-2, and PhyGenBench in both visual quality and benchmark-measured physical plausibility.

Problem

• physics plausibility
• video generation
• physical dynamics
• latent alignment
• real-world motion

Innovation

• Physics-Informed Generation
• Mixture-of-Experts
• Latent Alignment
• Flow-Matching Dynamics
• Physical Plausibility