Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

2026-07-31

AI Summary

This work addresses the challenges in multi-reward reinforcement learning, where conflicting objectives often lead to high alignment taxes, unstable training, and poor sample efficiency. To mitigate these issues, the authors propose PRISM, a novel framework that eschews conventional reward mixing by decomposing the policy space into multiple independent positive policies and a single global negative policy. These components are optimized separately and flexibly recombined during inference. This approach effectively alleviates objective conflicts, substantially improving training stability and efficiency while enabling on-demand control over behavioral preferences at deployment. Empirical results demonstrate that PRISM consistently outperforms existing methods across diverse tasks, including scientific reasoning, tool use, and helpfulness-safety alignment.

Abstract

Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases. As a result, multi-reward reinforcement learning (RL) has become an increasingly important problem for LLMs, where each reward captures a different aspect of desired behavior. However, optimizing with multiple rewards suffers from a more severe alignment tax issue, where different optimization objectives can trade off or even conflict with each other, leading to unstable and inefficient post-training. In this work, we propose PRISM, a new multi-reward RL framework built upon the idea of policy-space decomposition and composition. Instead of compositing different rewards, PRISM optimizes a set of standalone positive policies and a global negative policy. This alleviates the potential conflict during multi-reward policy optimization, while enabling controllability during inference by flexible policy composition. Experiments on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment show that PRISM consistently outperforms existing multi-reward RL baselines, with extra controllability for inference-time preference control.

Problem

• multi-reward RL
• alignment tax
• policy optimization
• reward conflict
• LLM alignment

Innovation

• policy decomposition
• multi-reward RL
• PRISM
• preference control
• policy composition