🤖 AI Summary
In text-to-image generation, a semantic gap exists between user prompts and model priors, yielding aesthetically pleasing yet semantically inaccurate images. To address this, we propose a training-free, vision-feedback-driven prompt optimization framework. Our method employs an automatic self-reflection module to localize missing concepts in generated images and leverages CLIP-based semantic analysis for fine-grained, goal-directed prompt revision. Designed as a plug-and-play module, it requires no model fine-tuning and is compatible with mainstream diffusion models. Evaluated on multiple semantic alignment benchmarks, our approach achieves state-of-the-art performance: it significantly improves content accuracy while preserving visual fidelity. The core innovations are (1) a training-free online self-reflection mechanism that dynamically identifies semantic deficiencies, and (2) a semantic–visual co-optimization paradigm that jointly refines prompts and image generation through cross-modal feedback.
📝 Abstract
Since there exists a notable gap between user-provided and model-preferred prompts, generating high-quality and satisfactory images using diffusion models often requires prompt engineering to optimize user inputs. Current studies on text-to-image prompt engineering can effectively enhance the style and aesthetics of generated images. However, they often neglect the semantic alignment between generated images and user descriptions, resulting in visually appealing but content-wise unsatisfying outputs. In this work, we propose VisualPrompter, a novel training-free prompt engineering framework that refines user inputs to model-preferred sentences. In particular, VisualPrompter utilizes an automatic self-reflection module to identify the missing concepts in generated images and a target-specific prompt optimization mechanism to revise the prompts in a fine-grained manner. Extensive experiments demonstrate the effectiveness of our VisualPrompter, which achieves new state-of-the-art performance on multiple benchmarks for text-image alignment evaluation. Additionally, our framework features a plug-and-play design, making it highly adaptable to various generative models.