Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation

2026-06-07

AI Summary

This work addresses the challenge that user-provided text prompts for text-to-image generation are often underspecified or ambiguous, leading models to deviate from intended outputs, while existing prompt refinement methods lack visual grounding and risk over-interpretation. To bridge this gap, the authors propose FaithRewriter, a novel framework that leverages an initial image generated by a multimodal large language model as a visual anchor to guide a large language model in producing semantically enriched yet visually consistent refined prompts. The optimized prompts are then distilled for deployment on lightweight models. By jointly enhancing semantic expressiveness and visual fidelity, FaithRewriter significantly outperforms existing baselines in both faithfulness to user intent and visual plausibility, effectively narrowing the discrepancy between user intention and generated imagery.

Abstract

Despite the impressive capabilities of text-to-image (T2I) models, an intent-generation gap often persists due to the brevity and ambiguity of user prompts. Existing approaches primarily polish the prompt for fluency and readability. However, the enhancement process still lacks visual grounding. As a result, the rewriter may over-infer missing details, causing an intent-generation gap. To address this limitation, we propose FaithRewriter, a novel prompt-enhancement framework for T2I generation. Specifically, FaithRewriter first leverages a multimodal MLLM to generate an image from the original prompt as an intermediate visual cue. This cue is then combined with the prompt and fed into a large-scale LLM to produce visually grounded augmentations that better reflect how the intended content should appear in images. Finally, these augmentations are distilled into a small-scale LLM for efficient deployment, enhancing its ability to generate effective T2I prompts. Experiments show that FaithRewriter yields prompts that are more faithful to the user intent and more visually plausible than strong baselines, helping narrow the intent-generation gap.

Problem

• text-to-image generation
• prompt ambiguity
• intent-generation gap
• visual grounding
• prompt rewriting

Innovation

• prompt rewriting
• visual grounding
• text-to-image generation
• multimodal LLM
• intent-generation gap