ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradients

2026-09-28

AI Summary

This study addresses the challenge of aligning learning signals with objectives in multi-reward policy optimization by proposing the ORPG method. Specifically, ORPG constructs independent clipping objectives for each reward and formulates the policy update as the unique solution to a spherical directional compromise. When gradients are compatible, it employs cosine-dependent interpolation for fusion; when conflicts arise, it performs gradient projection according to predefined priorities, thereby achieving synergistic updates within a single policy. Experimental results demonstrate that ORPG significantly outperforms baseline methods in both helpfulness–safety alignment and mathematical reasoning tasks involving correctness–cost trade-offs, effectively improving accuracy while reducing response length.

Abstract

Multi-reward policy optimization requires a joint update that reflects both the learning signals and the intended relationships among objectives. We introduce Objective-wise Reconciled Policy Gradient (ORPG), which constructs a separate clipped policy objective for each reward and reconciles the resulting gradients into one policy update. For compatible gradients, a cosine-dependent interpolation coordinates their contributions through a partially normalized reference while preserving the norm of their sum. We characterize this update as the unique solution of a spherical directional compromise. For conflicting gradients, projection follows the task's priorities. We evaluate the same compatible rule in helpfulness--safety alignment and correctness--cost optimization for mathematical reasoning. ORPG substantially improves average Useful and Harmless scores over the strongest external baseline on each axis. In mathematics, it achieves the highest average full-budget accuracy and three-budget hypervolume among the compared methods, with more accurate and shorter responses than the initial policy. Component comparisons and training dynamics show the larger contribution of compatible coordination and a complementary benefit from conflict handling. These results support gradient reconciliation for objectives with equal standing and for objectives with an explicit priority.

Problem

• multi-reward policy optimization
• gradient reconciliation
• multi-objective alignment
• conflicting gradients

Innovation

• Objective-wise Policy Gradients
• Gradient Reconciliation
• Multi-reward Optimization
• Cosine-dependent Interpolation
• Conflict Resolution