DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Masking

2026-07-20

AI Summary

This work addresses the challenge that existing video generation models often violate physical laws when handling complex non-rigid deformations due to insufficient physical reasoning about key dynamic regions. To overcome this limitation, the authors propose DeforM, a novel framework that introduces, for the first time, a vision-language model (VLM)-driven physical reasoning mechanism to automatically generate spatiotemporal masks that guide diffusion models to focus on regions with significant deformations. DeforM encompasses both a training-free strategy (DeforM-Free) and a training-dependent variant (DeforM-Injection), augmented with a physics-aware attention mechanism. Experimental results demonstrate that DeforM substantially outperforms current state-of-the-art methods in deformation-intensive scenarios, achieving marked improvements in both visual fidelity and physical consistency.

Abstract

Video generation models achieve high visual quality but often struggle to generate physics-aware videos. Unlike rigid-body motion, which can be described by explicit trajectories or formulas, complex deformation dynamics remain challenging to synthesize. We observe that a lack of physical reasoning for localizing dynamic areas allows irrelevant regions to dilute the model's attention, leading to generation failure. In this paper, we propose DeforM, a reasoning-guided image-to-video generation framework that directs the model's focus toward physics-critical regions. To reason about and localize these critical regions, we introduce a VLM-guided physical reasoning module, DeforM-Reason, to identify target objects and generate spatial-temporal masks. For physical guidance, we develop two alternative strategies: DeforM-Free for training-free mechanism analysis and DeforM-Injection as a powerful training-based generator. Experimental results demonstrate that DeforM improves the realism of generated deformation scenarios, outperforming baseline models in both visual quality and physical consistency.

Problem

• physics-aware video generation
• complex deformation dynamics
• spatial-temporal masking
• physical reasoning
• video realism

Innovation

• physics-aware video generation
• spatial-temporal masking
• visual language model (VLM)
• deformation dynamics
• reasoning-guided generation