Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback

2026-06-04

AI Summary

Current text-to-image models often generate defects that are local, subtle, and structurally complex, yet lack an instance-level feedback mechanism capable of simultaneously localizing defect locations, identifying types, attributing causes, and assessing their significance. This work formulates defect diagnosis as a structured set prediction task, representing each defect as a tuple of (location, type, cause, importance), and introduces the Structured Defect Grounding (SDG) framework. SDG transcends conventional heatmap regression paradigms by enabling, for the first time, semantic binding and interpretable diagnosis of multiple defects. Built upon a vision-language model, the SDG detector leverages the BoxFlow-GRPO algorithm to transform defect predictions into bounding-box-guided, importance-weighted spatial reward signals. Experiments demonstrate that the proposed method outperforms leading closed-source models in structured defect localization, yields rewards that substantially enhance diffusion model alignment, and facilitates localized image refinement.

Abstract

Despite generating increasingly photorealistic images, text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures. Diagnosing these failures requires instance-level feedback that answers where a defect occurs, what type it is, why it is defective, and its importance to overall image quality. While recent dense-feedback methods move beyond scalar supervision, their heatmap-centric representations still formulate diagnosis as pixel-field regression, making it difficult to localize variable-cardinality defects and bind semantic reasons to individual failures. To address this representation bottleneck, we propose Structured Defect Grounding (SDG), which casts T2I diagnosis as structured set prediction by modeling each defect as a (location, type, reason, importance) tuple. To make this formulation trainable and measurable, we introduce SDG-30K, a 30K-image dataset with box-grounded annotations across four modern T2I generators, together with a dedicated evaluation protocol, SDG-Eval. Building on this structured representation, we further present a diagnosis-to-alignment framework in which a Vision-Language Model (VLM) serves as the SDG detector, and BoxFlow-GRPO converts predicted defect sets into box-derived, importance-weighted spatial rewards for diffusion model alignment. Extensive experiments show that our SDG detector outperforms leading proprietary VLMs on structured defect grounding, while SDG-guided rewards consistently improve T2I alignment and support localized image refinement. These results establish SDG as a unified, instance-level interface for diagnosing, evaluating, and enhancing modern generative models.

Problem

• text-to-image
• defect diagnosis
• structured feedback
• instance-level grounding
• generative model alignment

Innovation

• Structured Defect Grounding
• Text-to-Image Diagnosis
• Set Prediction
• Vision-Language Model
• Diffusion Alignment