MatPhys: Learning Material-Aware Physics Parameters for Deformable Object Simulation from Videos

2026-05-19

AI Summary

Existing methods for reconstructing and simulating deformable objects from monocular videos typically assume homogeneous material properties and struggle to maintain consistent physical parameters for the same material across different scenes. This work proposes a material-aware feedforward framework that maps DINO visual features to physical parameters of a mass-spring system through semantic part decomposition and a shared material embedding codebook. By incorporating part-level material priors and leveraging a variational autoencoder architecture to associate appearance with physical attributes, the method achieves reconstruction accuracy and future state prediction performance on par with current baselines—without requiring per-scene optimization. Moreover, it demonstrates superior generalization to unseen objects and interactions, along with consistent cross-scene material modeling.

Abstract

Reconstructing simulation-ready deformable objects is important for vision, graphics, and robotics. Existing physics-driven methods can recover physical digital twins from videos, but they suffer from two fundamental limitations: they typically assume a homogeneous material across the whole object, and their scene-specific inverse optimization, combined with the inherent ambiguity of monocular observation, yields inconsistent parameters for the same material across different scenes or interactions. We propose MatPhys, a material-aware feed-forward framework that predicts spring-mass parameters from a single-view video, addressing these two issues with two coupled designs. To relax the homogeneous material assumption, we use DINO features to decompose the object into semantically meaningful parts and to query a part-level material prior, assigning each part its own physical behavior. To enforce cross-scene consistency, we introduce a learned material codebook of shared material embeddings as the bridge between appearance and physics, and further use the part-level prior as a reference distribution that constrains the decoder so that the same material yields consistent parameters across scenes and interactions. Together, these designs turn an under-constrained monocular problem into feed-forward inference grounded on shared, reusable material concepts. Experiments show that our method matches per-scene optimization baselines in reconstruction and future prediction, while achieving stronger generalization to unseen interactions and objects with more consistent physical parameters.

Problem

• deformable object simulation
• material-aware physics
• monocular video
• parameter consistency
• physical digital twin

Innovation

• material-aware simulation
• deformable object reconstruction
• physics parameter learning
• shared material codebook
• monocular video