GASP: Gaussian Splatting for Physic-Based Simulations

2024-09-09

arXiv.org

AI Summary

This work addresses the limitation of existing physics-based 3D simulation methods—where Gaussian splatting (GS) is coupled with dynamics only via explicit surface meshes (e.g., Marching Cubes)—by proposing the first mesh-free, differentiable physical modeling framework for GS. Methodologically, it treats 3D Gaussian primitives as discrete Newtonian point masses governed by physical dynamics; Gaussian distributions are flattened and deformed via triangle-mesh parameterization, enabling plug-and-play integration with arbitrary black-box physics engines. Key contributions include: (1) the first end-to-end differentiable, mesh-free Gaussian physical model; (2) a novel patch-driven deformation mechanism that jointly preserves geometric fidelity and dynamic consistency; and (3) significant improvements in visual quality and physical plausibility across multiple 3D object rendering benchmarks, while maintaining real-time rendering efficiency.

Technology Category

• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Physics simulation is paramount for modeling and utilization of 3D scenes in various real-world applications. However, its integration with state-of-the-art 3D scene rendering techniques such as Gaussian Splatting (GS) remains challenging. Existing models use additional meshing mechanisms, including triangle or tetrahedron meshing, marching cubes, or cage meshes. As an alternative, we can modify the physics grounded Newtonian dynamics to align with 3D Gaussian components. Current models take the first-order approximation of a deformation map, which locally approximates the dynamics by linear transformations. In contrast, our Gaussian Splatting for Physics-Based Simulations (GASP) model uses such a map (without any modifications) and flat Gaussian distributions, which are parameterized by three points (mesh faces). Subsequently, each 3D point (mesh face node) is treated as a discrete entity within a 3D space. Consequently, the problem of modeling Gaussian components is reduced to working with 3D points. Additionally, the information on mesh faces can be used to incorporate further properties into the physics model, facilitating the use of triangles. Resulting solution can be integrated into any physics engine that can be treated as a black box. As demonstrated in our studies, the proposed model exhibits superior performance on a diverse range of benchmark datasets designed for 3D object rendering.

Problem

• Integrating Gaussian Splatting with physics simulations for 3D scenes
• Replacing traditional meshing with flat Gaussian distributions
• Enhancing physics models using mesh face properties

Innovation

• Modifies Newtonian dynamics for Gaussian components
• Uses flat Gaussian distributions with three points
• Integrates with black box physics engines