Lightweight Optimal-Transport Harmonization on Edge Devices

2025-11-16

AI Summary

To address the unnatural appearance of augmented reality (AR) scenes caused by color inconsistency between virtual objects and real-world backgrounds, this paper proposes a lightweight, real-time color harmonization method. Grounded in optimal transport theory, our approach employs a compact encoder to directly predict the Monge–Kantorovich transport map for pixel-level color transfer. Notably, this is the first work to adapt optimal transport to on-device AR color harmonization, enabling efficient inference on edge devices. Our key contributions are: (1) the first pixel-accurate, manually annotated dataset specifically designed for AR color harmonization, along with an open-source toolkit for data acquisition; and (2) state-of-the-art performance on real AR composite images—achieving superior visual quality while maintaining real-time efficiency, thus attaining an optimal trade-off between fidelity and computational cost.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Color harmonization adjusts the colors of an inserted object so that it perceptually matches the surrounding image, resulting in a seamless composite. The harmonization problem naturally arises in augmented reality (AR), yet harmonization algorithms are not currently integrated into AR pipelines because real-time solutions are scarce. In this work, we address color harmonization for AR by proposing a lightweight approach that supports on-device inference. For this, we leverage classical optimal transport theory by training a compact encoder to predict the Monge-Kantorovich transport map. We benchmark our MKL-Harmonizer algorithm against state-of-the-art methods and demonstrate that for real composite AR images our method achieves the best aggregated score. We release our dedicated AR dataset of composite images with pixel-accurate masks and data-gathering toolkit to support further data acquisition by researchers.

Problem

• Achieving real-time color harmonization for augmented reality applications
• Developing lightweight on-device inference for seamless visual integration
• Creating optimal transport-based harmonization without computational bottlenecks

Innovation

• Lightweight encoder predicts optimal transport map
• On-device inference for real-time AR harmonization
• Compact model trained with classical optimal transport theory