Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency

2023-12-08

AI Summary

In image inpainting, generative models often produce spurious object hallucinations and chromatic inconsistencies under large, irregular masks. To address this, we propose ASUKA—a plug-and-play post-processing framework that significantly enhances realism without retraining the backbone model. First, we introduce a novel reconstruction prior grounded in Masked Autoencoders (MAE), which suppresses implausible object generation within masked regions. Second, we design a dedicated VAE decoder that explicitly models decoding as a local color harmonization task, enabling end-to-end optimization of color consistency. Evaluated on SD 1.5- and FLUX-based inpainting variants, ASUKA substantially reduces both object hallucination rates and color deviation metrics. It achieves state-of-the-art performance on Places2 and MISATO benchmarks, outperforming existing diffusion- and flow-based approaches. ASUKA establishes an efficient, modular, and training-free post-processing paradigm for generative inpainting.

Abstract

Recent advances in image inpainting increasingly use generative models to handle large irregular masks. However, these models can create unrealistic inpainted images due to two main issues: (1) Unwanted object insertion: Even with unmasked areas as context, generative models may still generate arbitrary objects in the masked region that don't align with the rest of the image. (2) Color inconsistency: Inpainted regions often have color shifts that causes a smeared appearance, reducing image quality. Retraining the generative model could help solve these issues, but it's costly since state-of-the-art latent-based diffusion and rectified flow models require a three-stage training process: training a VAE, training a generative U-Net or transformer, and fine-tuning for inpainting. Instead, this paper proposes a post-processing approach, dubbed as ASUKA (Aligned Stable inpainting with UnKnown Areas prior), to improve inpainting models. To address unwanted object insertion, we leverage a Masked Auto-Encoder (MAE) for reconstruction-based priors. This mitigates object hallucination while maintaining the model's generation capabilities. To address color inconsistency, we propose a specialized VAE decoder that treats latent-to-image decoding as a local harmonization task, significantly reducing color shifts for color-consistent inpainting. We validate ASUKA on SD 1.5 and FLUX inpainting variants with Places2 and MISATO, our proposed diverse collection of datasets. Results show that ASUKA mitigates object hallucination and improves color consistency over standard diffusion and rectified flow models and other inpainting methods.

Problem

• Preventing unwanted object insertion in inpainted images
• Reducing color inconsistency in inpainted regions
• Avoiding costly retraining of generative models

Innovation

• Uses MAE for reconstruction-based priors
• Employs specialized VAE decoder for harmonization
• Post-processing approach named ASUKA