Interactive Holographic Visualization for 3D Facial Avatar

2025-02-12

AI Summary

Traditional medical training relies on flat-screen displays or static models for visualizing dynamic facial expressions, resulting in limited realism and interactivity. To address this, we propose a real-time interactive holographic virtual human system specifically designed for pain assessment training. Our approach pioneers the integration of 3D Gaussian Splatting into viewpoint-adaptive holographic facial projection, synergistically combining real-time viewpoint calibration, stereoscopic holographic rendering, and interactive expression-driven facial modeling. This enables high-fidelity, dynamic, nonverbal feedback rendering from arbitrary viewing angles. To our knowledge, this is the first method to bridge the technical gap in dynamic, interactive 3D facial holography. Evaluated in medical simulation education, it achieves >30 FPS real-time performance while significantly improving medical students’ accuracy in recognizing pain-related facial expressions and enhancing training immersion.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Computer Vision: 3D Computer Vision
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Traditional methods for visualizing dynamic human expressions, particularly in medical training, often rely on flat-screen displays or static mannequins, which have proven inefficient for realistic simulation. In response, we propose a platform that leverages a 3D interactive facial avatar capable of displaying non-verbal feedback, including pain signals. This avatar is projected onto a stereoscopic, view-dependent 3D display, offering a more immersive and realistic simulated patient experience for pain assessment practice. However, there is no existing solution that dynamically predicts and projects interactive 3D facial avatars in real-time. To overcome this, we emphasize the need for a 3D display projection system that can project the facial avatar holographically, allowing users to interact with the avatar from any viewpoint. By incorporating 3D Gaussian Splatting (3DGS) and real-time view-dependent calibration, we significantly improve the training environment for accurate pain recognition and assessment.

Problem

• Dynamic 3D facial avatar visualization
• Real-time interactive holographic projection
• Enhanced pain assessment training environment

Innovation

• 3D interactive facial avatar
• holographic 3D display
• real-time view-dependent calibration