Training Language Model to Critique for Better Refinement

2025-06-27

AI Summary

Existing work lacks a systematic investigation into which types of critique most effectively improve model response quality. To address this, we propose Refinement-oriented Critique Optimization (RCO), a critique optimization framework explicitly designed to enhance subsequent responses. RCO introduces Critique Utility (CU)—a learnable, automated reward signal that quantifies the actual improvement a critique induces in the refined response. It establishes a critique–refinement closed loop, integrating reinforcement learning with fine-grained preference supervision to enable iterative training without human-annotated preferences. Empirically, RCO significantly improves the instructiveness and practical utility of critique models. It consistently outperforms strong baselines across five diverse tasks—including dialogue generation, summarization, and question answering—while simultaneously enhancing both critique quality and the magnitude of response improvement.

Technology Category

• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce extbf{R}efinement-oriented extbf{C}ritique extbf{O}ptimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method's effectiveness in enhancing LLM critique-refinement loops.

Problem

• Optimizing critique types for better model response refinement
• Training critic models using refinement signals effectively
• Enhancing critique quality across multiple LLM tasks

Innovation

• Refinement-oriented Critique Optimization framework
• Feedback loop with critique utility reward
• Training critic models via refinement signals