SiPhy: Single-Image Physical Property Reasoning

2026-07-24

AI Summary

This work proposes SiPhy, the first unified framework capable of jointly estimating physical properties—such as mass, density, and Young’s modulus—from a single RGB image without relying on multi-view reconstruction or physics-based simulation supervision. The method integrates 3D-aware visual cues, depth information, and language-guided material knowledge through several novel components: pseudo-voxel sampling, vision–language material alignment, and gravity-aware volume optimization. Leveraging CLIP features and vision–language models, it generates plausible material candidates and refines estimates via part-wise contrastive aggregation and gravity-aware refinement. Evaluated across multiple benchmarks, SiPhy substantially outperforms existing approaches, achieving a 93% improvement in mass MnRE, a 35.5% reduction in density MAE, and a 23.5% decrease in Young’s modulus error, while enabling automatic annotation of real-world hand–object interaction data.

Abstract

Inferring physical properties such as mass, stiffness, and elasticity from a single image is essential for simulation and embodied AI, yet most existing approaches rely on multi-view reconstruction or physics-based supervision. We introduce SiPhy, a unified framework for single-image physical property reasoning that aligns 3D-aware visual cues, depth with language-based material knowledge. From one RGB image, SiPhy samples pseudo-voxel points, extracts CLIP features, and grounds them to material candidates proposed by a VLM. A part-based contrastive aggregator enforces region consistency, while a heaviness-aware refinement improves thickness and volume estimation for dense objects. Across ABO-500, MVImgNet-100, and PhysXNet-100, SiPhy achieves state-of-the-art single-image performance, surpassing multi-view reconstruction methods by improving mass MnRE by up to 93% (vs. PUGS), reducing density MAE by 35.5% (vs. NeRF2Physics), and lowering Young's modulus error by 23.5%. We further validate SiPhy on real hand-object interaction datasets, demonstrating its potential as a data annotation engine for physical understanding from single-view imagery.

Problem

• physical property inference
• single-image reasoning
• mass estimation
• material understanding
• embodied AI

Innovation

• single-image physical reasoning
• 3D-aware visual cues
• material grounding
• part-based contrastive aggregation
• heaviness-aware refinement