Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting

2024-11-19

arXiv.org

AI Summary

To address the high computational cost and manual material specification requirements in physics-based simulation for open-world static 3D scenes, this paper proposes MLLM-P3: a novel framework introducing zero-shot physical property perception via multimodal large language models (MLLMs), coupled with geometry-adaptive particle sampling (PGAS). The method reformulates physics modeling as material property distribution prediction and probabilistic dynamics solving. Integrating Gaussian splatting reconstruction, open-vocabulary 3D segmentation, multi-view inpainting, and material property distribution prediction (MPDP), MLLM-P3 enables fully automatic, parameter-free interactive physical simulation. On a single GPU, it generates high-fidelity motion sequences in just two minutes—surpassing state-of-the-art methods in motion realism. A user study confirms significant improvements in perceived naturalness and interactive utility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in 3D generation models have opened new possibilities for simulating dynamic 3D object movements and customizing behaviors, yet creating this content remains challenging. Current methods often require manual assignment of precise physical properties for simulations or rely on video generation models to predict them, which is computationally intensive. In this paper, we rethink the usage of multi-modal large language model (MLLM) in physics-based simulation, and present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics. We begin with detailed scene reconstruction and object-level 3D open-vocabulary segmentation, progressing to multi-view image in-painting. Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception (MLLM-P3) to predict mean physical properties of objects in a zero-shot manner. Based on the mean values and the object's geometry, the Material Property Distribution Prediction model (MPDP) model then estimates the full distribution, reformulating the problem as probability distribution estimation to reduce computational costs. Finally, we simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling (PGAS) strategy, efficiently capturing complex deformations and significantly reducing computational costs. Extensive experiments and user studies demonstrate our Sim Anything achieves more realistic motion than state-of-the-art methods within 2 minutes on a single GPU.

Problem

• Simulating dynamic 3D object movements efficiently
• Predicting physical properties without manual input
• Reducing computational costs in physics-based simulations

Innovation

• MLLM-based Physical Property Perception predicts properties.
• Material Property Distribution Prediction reduces computational costs.
• Physical-Geometric Adaptive Sampling captures complex deformations efficiently.