Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
This study addresses the signal imbalance in multi-reward reinforcement learning caused by uneven reward activation densities, which constrains the effectiveness of multi-objective training for large language models. To this end, it proposes a density-aware reward aggregation mechanism that reveals the intrinsic relationship between advantage energy and activation density. Specifically, the method dynamically adjusts the weights of sparse rewards through inverse square root density correction. Furthermore, by integrating GDPO normalization, it introduces an adaptive weighting algorithm that requires no modifications to the underlying objectives. Experimental results demonstrate that this approach reduces the number of training steps by 26% on tool-calling tasks and significantly improves length compliance in mathematical reasoning, while maintaining competitive overall performance.