Seeing the Wind from a Falling Leaf

📅 2025-11-30
📈 Citations: 0
Influential: 0
📄 PDF
🤖 AI Summary
This work addresses the inverse problem of inferring implicit physical force fields—such as time-varying wind fields—that drive object motion from monocular video. We propose an end-to-end differentiable inverse graphics framework that jointly optimizes object geometry, material properties, and external force field parameters. By tightly coupling differentiable physics simulation with neural rendering, our method enables gradient-based inversion from observed motion to physically plausible force fields. Evaluated on both synthetic and real-world falling-leaf videos, our approach successfully recovers temporally coherent, physically consistent wind fields. Moreover, the inferred force fields support dynamics-aware video generation and editing. Our key contribution is the first differentiable, joint, end-to-end visual-to-force inversion framework—bridging perception and physics without explicit force supervision—while preserving dynamical consistency throughout inference and synthesis.

Technology Category

Application Category

📝 Abstract
A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. We hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap between vision and physics. Please check more video results in our href{https://chaoren2357.github.io/seeingthewind/}{project page}.
Problem

Research questions and friction points this paper is trying to address.

Recover invisible forces from visual observations
Estimate wind fields by observing object motions
Bridge vision and physics through inverse graphics
Innovation

Methods, ideas, or system contributions that make the work stand out.

End-to-end differentiable inverse graphics framework
Recovers invisible forces from object motion videos
Models geometry, physics, and interactions jointly
🔎 Similar Papers
No similar papers found.
Z
Zhiyuan Gao
University of Southern California
Jiageng Mao
Jiageng Mao
University of Southern California
RoboticsComputer Vision
H
Hong-Xing Yu
Stanford University
Haozhe Lou
Haozhe Lou
University of Southern California
RoboticControl systemPhysics informed neural networkDifferentiable Simulator
E
Emily Yue-Ting Jia
University of Southern California
Jernej Barbic
Jernej Barbic
University of Southern California
J
Jiajun Wu
Stanford University
Y
Yue Wang
University of Southern California