Group Entropy-Controlled Policy Optimization

2026-07-18

AI Summary

This work addresses the challenge of inter-task entropy heterogeneity in multi-task reinforcement learning alignment with large language models, where uniform or token-level entropy regularization fails to accommodate diverse exploration needs across tasks, and GRPO-style methods suffer from biased advantage signals due to entropy disparities. To resolve this, the paper proposes GEPO, a novel group-level entropy-aware advantage shaping mechanism that asymmetrically adjusts the advantage function based on historical entropy statistics of task groups: it attenuates positive advantages for low-entropy groups to curb over-exploitation and dampens negative advantages for high-entropy groups to sustain exploration. Integrated within the GRPO framework with grouped sampling, historical entropy estimation, and adaptive threshold clipping, GEPO achieves task-adaptive exploration–exploitation balance. Evaluated on 13 benchmarks spanning mathematics, physics, science, code generation, and instruction following, GEPO consistently outperforms GRPO and existing entropy control methods, delivering balanced and robust cross-task performance gains.

Abstract

Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.

Problem

• entropy control
• reinforcement learning
• heterogeneous tasks
• advantage bias
• large language models

Innovation

• Group Entropy
• Policy Optimization
• Advantage Shaping
• Reinforcement Learning
• Large Language Models