Dual Inversion for Text-to-Image Diffusion Models: From Both Prompt and Noise Perspectives

2026-07-29

AI Summary

Existing prompt inversion methods for text-to-image diffusion models struggle to simultaneously preserve image fidelity and semantic interpretability, often overlooking the critical role of latent noise in structural consistency. This work proposes Dualin, a two-stage joint inversion framework: the first stage integrates vision-language and large language models to generate faithful and human-readable hard prompts, while the second stage employs unconditional DDIM inversion to accurately recover latent noise, achieving dual alignment in both semantics and structure. Dualin is the first approach to unify prompt inversion with latent noise reconstruction, and we theoretically demonstrate that the recovered noise enables flexible editing without re-optimization, overcoming the limitations of prompt-only paradigms. Experiments show that Dualin achieves high-quality prompt inversion and state-of-the-art image fidelity across multiple datasets, laying a foundation for precise and controllable image editing.

Abstract

Prompt inversion, as a typical reverse engineering technique, enables text-to-image (T2I) diffusion models to generate the desired target images without extensive prompt engineering. However, existing prompt inversion methods suffer from significant limitations: (1) gradient-based methods are unstable and uninterpretable, often resulting in generated images with severe artifacts; (2) gradient-free methods yield human-readable prompts but still fail to preserve visual fidelity due to the lack of fine-grained detail alignment. We contend that the limitations stem from treating prompt inversion as a sufficient condition for reverse engineering, ignoring the critical role of the latent noise that encodes structural information. Consequently, we propose Dualin (Dual inversion), a two-stage method that jointly recovers both the semantic prompt and latent noise of the target image. In the first stage, we integrate vision-language model, CLIP and large language model to invert a faithful, human-interpretable hard prompt. In the second stage, unconditional DDIM inversion reconstructs the exact latent noise of the target image, guaranteeing the consistency at the structural information level. Theoretically, we prove that the inverted noise enables flexible image editing without re-optimization. Extensive experiments on diverse datasets demonstrate that Dualin simultaneously generates high-quality inverted prompts and achieves state-of-the-art image fidelity. Additionally, Dualin can establish a robust foundation for the precise and controllable image editing.

Problem

• prompt inversion
• text-to-image diffusion models
• latent noise
• image fidelity
• reverse engineering

Innovation

• prompt inversion
• latent noise
• dual inversion
• diffusion models
• image fidelity