PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

2026-08-05

AI Summary

Current vision-language models exhibit limited capabilities in physical reasoning over videos, particularly in future prediction and counterfactual inference. This work proposes the first training-free agent framework that automatically constructs a reusable and editable implicit physical world from video by integrating object segmentation, mesh reconstruction, 6D pose tracking, and continuous-time dynamics modeling, enabling precise intervention-based simulation. Departing from conventional discrete-time-step simulation paradigms, the method achieves performance gains of 38.23 and 8.08 percentage points on CLEVRER and Physion++, respectively, and surpasses the strongest existing vision-language model baseline, GPT-5.5, by 19.25 points in counterfactual reasoning.

Abstract

Reliable physical reasoning from video requires understanding how objects move, interact, and respond to interventions. Existing vision-language models (VLMs) often struggle to interpret these dynamics and reason reliably about future and counterfactual outcomes. We introduce PhysMind, a training-free agentic framework that constructs one reusable, question-agnostic executable world per video. PhysMind recovers a temporally consistent dynamic scene through object segmentation, mesh reconstruction, and 6D pose tracking, then fits analytic continuous-time dynamics and latent physical parameters without unrolling a time-stepped simulator. Given a question, it inspects, continues, or edits the world and answers from the resulting trajectories and interactions. Relative to direct chain-of-thought (CoT) reasoning with the same VLM, PhysMind improves accuracy by 38.23 points on CLEVRER and 8.08 points on Physion++. On counterfactual questions, it exceeds the strongest evaluated VLM baseline, GPT-5.5, by 19.25 points.

Problem

• physical reasoning
• video understanding
• counterfactual reasoning
• object dynamics
• vision-language models

Innovation

• training-free
• executable world
• physical reasoning
• continuous-time dynamics
• counterfactual reasoning