PRISM: Prompt Refinement via Image-grounded Self-rewarding Mechanism for Text-to-Image Generation

2026-07-27

AI Summary

This work addresses the high sensitivity of current text-to-image generation models to prompt wording and the absence of diagnostic feedback grounded in generated images, as well as reusable optimization strategies. To bridge this gap, we propose PRISM, a novel framework that establishes a closed-loop “prompt–image–feedback” pipeline by introducing, for the first time, image-driven structured diagnostics and a hybrid ideal-point–Chebyshev self-reward mechanism. PRISM leverages a unified vision-language model fine-tuned on multiple tasks to deliver interpretable scores along three dimensions—semantic consistency, aesthetic quality, and human preference—and iteratively refines prompts through self-reward reinforcement learning. Experimental results demonstrate that PRISM significantly enhances both overall image quality and fine-grained semantic alignment, while yielding reusable and interpretable prompt optimization pathways.

Abstract

Text-to-image generation models can synthesize high-quality images from natural language descriptions, but their performance remains highly sensitive to prompt formulation. Existing prompt optimization methods mainly rely on text-side rewriting, prompt expansion, or external reward signals, offering limited image-grounded diagnosis and weak support for learning reusable optimisation policies. In this paper, we propose PRISM, a Prompt Refinement framework via Image-grounded Self-rewarding Mechanism. PRISM closes the prompt-image-feedback loop by interpreting generated images with structured visual diagnosis and scoring them along semantic consistency, aesthetic quality, and human preference alignment. It first initializes a unified VLM through multi-task supervised fine-tuning, and then improves the prompt policy via self-rewarding optimization with a hybrid ideal-point and Chebyshev reward. Extensive experiments show that PRISM improves holistic image quality and fine-grained semantic alignment, while providing interpretable feedback for targeted prompt refinement. The code is available at https://anonymous.4open.science/r/PRISM-FF81.

Problem

• text-to-image generation
• prompt optimization
• image-grounded feedback
• semantic alignment
• prompt refinement

Innovation

• prompt refinement
• image-grounded feedback
• self-rewarding mechanism
• visual language model
• text-to-image generation