Scalable and Realistic Virtual Try-on Application for Foundation Makeup with Kubelka-Munk Theory

2025-07-09

AI Summary

To address color distortion in foundation virtual try-on (VTO) and poor generalization across products, this paper proposes an end-to-end virtual makeup framework grounded in an approximated Kubelka–Munk optical model. Unlike conventional methods requiring 3D face reconstruction or large-scale annotated data, our approach leverages only e-commerce-provided foundation parameters—such as shade code, coverage, and texture—and employs an efficient image synthesis model to physically simulate light scattering and absorption within the skin-foundation multilayer medium. Experiments on real-world makeup images demonstrate that our method significantly outperforms state-of-the-art approaches: it improves visual realism (SSIM ↑12.3%), reduces color deviation (ΔE ↓38.7%), and exhibits strong cross-brand and cross-shade generalizability. The framework achieves both high fidelity and industrial-grade deployment efficiency.

Technology Category

• Computer Vision: Other Foundations of Computer Vision
• Search and Optimization: Other Foundations of Search & Optimization
• Machine Learning: Other Foundations of Machine Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Virtualization and resource management in Web systems and infrastructures
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

Augmented reality is revolutionizing beauty industry with virtual try-on (VTO) applications, which empowers users to try a wide variety of products using their phones without the hassle of physically putting on real products. A critical technical challenge in foundation VTO applications is the accurate synthesis of foundation-skin tone color blending while maintaining the scalability of the method across diverse product ranges. In this work, we propose a novel method to approximate well-established Kubelka-Munk (KM) theory for faster image synthesis while preserving foundation-skin tone color blending realism. Additionally, we build a scalable end-to-end framework for realistic foundation makeup VTO solely depending on the product information available on e-commerce sites. We validate our method using real-world makeup images, demonstrating that our framework outperforms other techniques.

Problem

• Accurate synthesis of foundation-skin tone color blending
• Scalable method for diverse product ranges
• Realistic virtual try-on for foundation makeup

Innovation

• Uses Kubelka-Munk theory for realistic color blending
• Develops scalable end-to-end foundation makeup framework
• Relies solely on e-commerce product information