Rewarding Better Thinking for LLM Preference Alignment

2026-07-22

AI Summary

Current preference alignment methods for large language models predominantly rely on outcome-level rewards, which provide insufficient fine-grained guidance over the reasoning process and consequently lead to inadequate trajectory-level preference modeling. To address this limitation, this work proposes Thinking Checklist Reward (TCR), which introduces sample-specific thinking checklists to reformulate preference alignment as an evaluation of how well key reasoning considerations are covered throughout the inference trajectory. TCR further incorporates an exponential moving average (EMA) residual mechanism to effectively disentangle reasoning quality from final output quality, thereby isolating a “reasoning gain” that transcends outcome-based rewards. Experiments across five models spanning three model families demonstrate consistent improvements in alignment performance on multiple benchmarks, while ablation studies confirm the critical contributions of both checklist supervision and the EMA residual design.

Abstract

LLM preference alignment aims to optimize models toward human preferences across diverse user instructions. Reinforcement learning has become a major post-training approach for this goal, but existing proxy rewards are often outcome-level, mainly evaluating the final response while providing limited guidance for the reasoning trajectory. This can make credit assignment coarse when multiple responses receive similar final scores, leaving trajectory-level preferences under-specified. To address this limitation, we propose Thinking Checklist Reward (TCR), a process-oriented reward for RL-based preference alignment. TCR converts preference pairs into sample-specific thinking checklists and uses them to evaluate whether the generated reasoning trace addresses the preference-implied considerations. To reduce overlap with outcome-level supervision, TCR further introduces an exponential moving average (EMA) residual formulation to isolate a complementary thinking surplus beyond what is predictable from the outcome reward. Experiments on five models from three model families show that TCR consistently improves alignment performance across diverse benchmarks, with ablations further validating the importance of EMA-based residual formulation and sample-specific checklist supervision.

Problem

• preference alignment
• reasoning trajectory
• reward modeling
• credit assignment
• process-oriented reward

Innovation

• Thinking Checklist Reward
• process-oriented reward
• preference alignment
• reasoning trajectory
• EMA residual