PhysNote: Self-Knowledge Notes for Evolvable Physical Reasoning in Vision-Language Model

2026-04-27

AI Summary

This work addresses the failure of vision-language models in physical reasoning within dynamic real-world scenarios, where spatiotemporal identity drift and the transient nature of intermediate reasoning insights undermine performance. To mitigate these issues, the authors propose an agent framework that autonomously generates “knowledge notes” to externalize, reuse, and iteratively refine physical knowledge. The framework employs a spatiotemporal normalization mechanism to align object identities across time and space and incorporates a hierarchical knowledge base to support structured reasoning. By integrating vision-language models with a closed-loop reasoning process, the method achieves a 56.68% accuracy on PhysBench, surpassing the strongest multi-agent baseline by 4.96% and demonstrating consistent performance gains across four distinct domains of physical reasoning.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Low Level & Physics-based Vision
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance on textbook-style physics problems, yet they frequently fail when confronted with dynamic real-world scenarios that require temporal consistency and causal reasoning across frames. We identify two fundamental challenges underlying these failures: (1) spatio-temporal identity drift, where objects lose their physical identity across successive frames and break causal chains, and (2) volatility of inference-time insights, where a model may occasionally produce correct physical reasoning but never consolidates it for future reuse. To address these challenges, we propose PhysNote, an agentic framework that enables VLMs to externalize and refine physical knowledge through self-generated "Knowledge Notes." PhysNote stabilizes dynamic perception through spatio-temporal canonicalization, organizes self-generated insights into a hierarchical knowledge repository, and drives an iterative reasoning loop that grounds hypotheses in visual evidence before consolidating verified knowledge. Experiments on PhysBench demonstrate that PhysNote achieves 56.68% overall accuracy, a 4.96% improvement over the best multi-agent baseline, with consistent gains across all four physical reasoning domains.

Problem

• spatio-temporal identity drift
• causal reasoning
• physical reasoning
• vision-language models
• temporal consistency

Innovation

• PhysNote
• spatio-temporal canonicalization
• Knowledge Notes
• physical reasoning
• vision-language models