Reflections on Diversity: A Real-time Virtual Mirror for Inclusive 3D Face Transformations

2025-03-25

AI Summary

This study addresses the limited demographic diversity and anatomical plausibility of mainstream AR filters. To this end, we propose MOD, a real-time virtual mirror system. Methodologically, MOD integrates GAN-based texture editing with geometric control via 3D Morphable Models (3DMMs), establishing the first real-time, cross-gender and cross-ethnic 3D facial transformation framework that jointly ensures photorealism, demographic diversity, and anatomical consistency. We introduce the novel concept of “collective face,” enabling multi-face averaging to visualize population-level facial commonalities. Furthermore, we design a multidimensional evaluation protocol combining subjective user surveys with objective metrics—including CNN-based classification accuracy and perceptual realism scores. Experimental results demonstrate that MOD significantly outperforms commercial filters (e.g., Snapchat, TikTok) in ethnic/gender label accuracy (92.4%), perceived realism, and user-perceived respectfulness (87% approval rate).

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

Real-time 3D face manipulation has significant applications in virtual reality, social media and human-computer interaction. This paper introduces a novel system, which we call Mirror of Diversity (MOD), that combines Generative Adversarial Networks (GANs) for texture manipulation and 3D Morphable Models (3DMMs) for facial geometry to achieve realistic face transformations that reflect various demographic characteristics, emphasizing the beauty of diversity and the universality of human features. As participants sit in front of a computer monitor with a camera positioned above, their facial characteristics are captured in real time and can further alter their digital face reconstruction with transformations reflecting different demographic characteristics, such as gender and ethnicity (e.g., a person from Africa, Asia, Europe). Another feature of our system, which we call Collective Face, generates an averaged face representation from multiple participants' facial data. A comprehensive evaluation protocol is implemented to assess the realism and demographic accuracy of the transformations. Qualitative feedback is gathered through participant questionnaires, which include comparisons of MOD transformations with similar filters on platforms like Snapchat and TikTok. Additionally, quantitative analysis is conducted using a pretrained Convolutional Neural Network that predicts gender and ethnicity, to validate the accuracy of demographic transformations.

Problem

• Real-time 3D face transformation for diversity reflection
• Combining GANs and 3DMMs for realistic demographic changes
• Evaluating realism and accuracy of demographic face transformations

Innovation

• Combines GANs and 3DMMs for realistic face transformations
• Real-time facial capture and demographic feature alteration
• Generates averaged face from multiple participants' data