MPMAvatar: Learning 3D Gaussian Avatars with Accurate and Robust Physics-Based Dynamics

2025-10-01

AI Summary

Existing methods struggle to generate physically realistic, robust, and high-fidelity 3D human avatars wearing loose clothing from multi-view video input. This paper proposes a novel framework integrating Material Point Method (MPM)-based physics simulation with 3D Gaussian splatting rendering. We introduce an anisotropic elastic constitutive model to improve cloth mechanical fidelity, design an efficient collision handling algorithm for enhanced numerical stability, and incorporate data-driven dynamical priors to enable, for the first time, zero-shot synthesis of realistic garment dynamics for unseen interactive motions. Our approach comprehensively outperforms state-of-the-art methods in dynamic accuracy, free-viewpoint rendering quality, and cross-motion generalization—while maintaining computational efficiency and supporting complex interactive scene simulation.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Computer Vision: Low Level & Physics-based Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While there has been significant progress in the field of 3D avatar creation from visual observations, modeling physically plausible dynamics of humans with loose garments remains a challenging problem. Although a few existing works address this problem by leveraging physical simulation, they suffer from limited accuracy or robustness to novel animation inputs. In this work, we present MPMAvatar, a framework for creating 3D human avatars from multi-view videos that supports highly realistic, robust animation, as well as photorealistic rendering from free viewpoints. For accurate and robust dynamics modeling, our key idea is to use a Material Point Method-based simulator, which we carefully tailor to model garments with complex deformations and contact with the underlying body by incorporating an anisotropic constitutive model and a novel collision handling algorithm. We combine this dynamics modeling scheme with our canonical avatar that can be rendered using 3D Gaussian Splatting with quasi-shadowing, enabling high-fidelity rendering for physically realistic animations. In our experiments, we demonstrate that MPMAvatar significantly outperforms the existing state-of-the-art physics-based avatar in terms of (1) dynamics modeling accuracy, (2) rendering accuracy, and (3) robustness and efficiency. Additionally, we present a novel application in which our avatar generalizes to unseen interactions in a zero-shot manner-which was not achievable with previous learning-based methods due to their limited simulation generalizability. Our project page is at: https://KAISTChangmin.github.io/MPMAvatar/

Problem

• Modeling physically plausible dynamics of humans with loose garments
• Achieving accurate and robust physics-based avatar animation
• Enabling photorealistic rendering from free viewpoints

Innovation

• Uses Material Point Method for garment dynamics
• Incorporates anisotropic model and collision algorithm
• Combines physics with 3D Gaussian Splatting rendering