MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following

2026-06-04

AI Summary

This work addresses the instability of standard GRPO when training multi-constraint instruction-following models under discrete, low-variance rewards, where intra-group reward homogenization undermines z-score normalization. The study formally characterizes three pathological issues inherent to this setting and introduces a stable training framework that enhances reward diversity via multi-temperature sampling, restores gradients for homogeneous groups through a dual-anchor advantage function, incorporates prospect-theory-inspired update clipping and violation penalties, and employs asymmetric KL regularization. Evaluated on FollowBench, IFEval, and a newly curated multi-constraint dataset, the approach improves constraint satisfaction by up to 5.0% on Llama-3.2-3B, enables stable small-batch convergence, and preserves general capabilities as measured by MMLU and ARC benchmarks.

Abstract

Reinforcement learning with verifiable rewards is ideal for multi-constraint instruction following, yet standard group-relative policy optimization (GRPO) becomes unstable under discrete, low-dispersion rewards, where within-group reward distributions are frequently homogeneous. We identify and formalize three pathologies of z-score group normalization in this regime: low-variance amplification, mean-centering blindness, and zero-variance collapse. To address them, we propose MDP-GRPO, which stabilizes learning through (1) multi-temperature sampling to increase reward dispersion, (2) dual-anchor advantages to restore gradients in homogeneous groups and stop mean-centering blindness, (3) prospect-theoretic shaping to bound updates and penalize violations based on Kahneman and Tversky's theory, and (4) asymmetric KL regularization. Evaluated on FollowBench, IFEval, and a curated multi-constraint dataset, MDP-GRPO outperforms standard GRPO, improving strict constraint satisfaction by up to 5.0% on Llama-3.2-3B. Our method also enables stable convergence with small group sizes while preserving general capabilities on MMLU and ARC.

Problem

• multi-constraint instruction following
• group-relative policy optimization
• low-dispersion rewards
• reward homogeneity
• reinforcement learning instability

Innovation

• Group Relative Policy Optimization
• Multi-Constraint Instruction Following
• Reward Shaping
• Prospect Theory
• Asymmetric KL Regularization