Think Before You Score: Thinking Reward Model for Visual Generation

2026-09-29

AI Summary

This study addresses the limitation of existing visual reward models, which predominantly rely on implicit evaluation and lack adaptive criteria. We pioneer a "think-then-score" paradigm by introducing the Thinking Reward Model, which leverages the reasoning capabilities of large language models to first formulate instance-specific scoring rubrics before generating fine-grained, point-level rewards for guiding visual generation optimization. Furthermore, we propose the PD-GRPO algorithm to mitigate score polarization, effectively reconciling pairwise preference supervision with fine-grained scoring. By integrating reinforcement learning with fine-grained reward modeling, our approach achieves state-of-the-art performance among open-source models and substantially enhances the reinforcement training of diverse visual generation models.

Abstract

Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.

Problem

• visual reward model
• visual generation evaluation
• score polarization
• fine-grained scoring

Innovation

• Thinking Reward Model
• Case-adaptive Rubrics
• Pairwise Dual-Group Relative Policy Optimization
• Visual Generation
• Fine-grained Pointwise Rewards