Aligned Stable Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistency

2026-01-21

AI Summary

This work addresses the prevalent issues of object hallucination and color inconsistency in generative image inpainting, which often arise from contextual mismatches. To mitigate these problems, the authors propose ASUKA, a lightweight post-processing framework that uniquely integrates two mechanisms: a reconstruction prior for unknown regions to suppress the generation of irrelevant objects, and a localized color harmonization module implemented via a customized VAE decoder to ensure chromatic coherence. ASUKA is model-agnostic, seamlessly compatible with pre-trained inpainting models based on both U-Net and DiT architectures, and applicable to both diffusion and rectified flow paradigms. Extensive experiments on Places2 and the newly introduced MISATO benchmark demonstrate that ASUKA significantly reduces object hallucination and enhances color consistency, outperforming current state-of-the-art methods.

Abstract

Generative image inpainting can produce realistic, high-fidelity results even with large, irregular masks. However, existing methods still face key issues that make inpainted images look unnatural. In this paper, we identify two main problems: (1) Unwanted object insertion: generative models may hallucinate arbitrary objects in the masked region that do not match the surrounding context. (2) Color inconsistency: inpainted regions often exhibit noticeable color shifts, leading to smeared textures and degraded image quality. We analyze the underlying causes of these issues and propose efficient post-hoc solutions for pre-trained inpainting models. Specifically, we introduce the principled framework of Aligned Stable inpainting with UnKnown Areas prior (ASUKA). To reduce unwanted object insertion, we use reconstruction-based priors to guide the generative model, suppressing hallucinated objects while preserving generative flexibility. To address color inconsistency, we design a specialized VAE decoder that formulates latent-to-image decoding as a local harmonization task. This design significantly reduces color shifts and produces more color-consistent results. We implement ASUKA on two representative inpainting architectures: a U-Net-based model and a DiT-based model. We analyze and propose lightweight injection strategies that minimize interference with the model's original generation capacity while ensuring the mitigation of the two issues. We evaluate ASUKA using the Places2 dataset and MISATO, our proposed diverse benchmark. Experiments show that ASUKA effectively suppresses object hallucination and improves color consistency, outperforming standard diffusion, rectified flow models, and other inpainting methods. Dataset, models and codes will be released in github.

Problem

• unwanted object insertion
• color inconsistency
• image inpainting
• artifact hallucination
• texture smearing

Innovation

• image inpainting
• object hallucination suppression
• color consistency
• reconstruction prior
• VAE decoder