CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning

2026-09-29

AI Summary

This study addresses the issue in multi-reward Group Relative Policy Optimization (GRPO) training where reward scale discrepancies cause large-scale signals to dominate advantage normalization, thereby suppressing smaller-scale signals. To mitigate this, we propose CorrGRPO, which introduces a correlation-based normalization mechanism that transforms pairwise covariances into Pearson correlation coefficients. This innovation effectively decouples reward magnitudes from policy update magnitudes, enabling advantage estimation to adapt to the inter-reward correlation structure rather than absolute scales, thus achieving balanced gradient optimization across multiple objectives. Extensive experiments on code generation, tool calling, and agent safety tasks demonstrate that CorrGRPO yields significant performance improvements for models ranging from 0.5B to 8B parameters.

Abstract

Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across three domains, including code generation, tool calling, and agent security. Our code is available at https://github.com/HKUST-KnowComp/CorrGRPO.

Problem

• Multi-Reward Learning
• Group Relative Policy Optimization
• Reward Normalization
• Correlated Rewards
• Reasoning Language Models

Innovation

• Group Relative Policy Optimization
• Multi-Reward Learning
• Pearson Correlation Normalization
• Advantage Estimation
• Reinforcement Learning