reward design

This paper addresses the misalignment between reward models and true objectives in deep reinforcement learning, as well as the resulting limitations in policy optimization. To this end, it introduces— for the first time—a unified taxonomy that systematically organizes reward modeling across three orthogonal dimensions: modeling source (explicit vs. implicit), mechanism design (supervised vs. interactive), and learning paradigm (static vs. dynamic). The survey comprehensively covers mainstream approaches—including inverse reinforcement learning, preference learning, language-model-based feedback, human demonstration distillation, contrastive learning, and online interactive modeling—and critically analyzes evaluation methodologies and practical deployment challenges. This work fills a critical gap in the literature by providing the first systematic, cross-cutting review of reward modeling. It clarifies the technical evolution of the field and identifies four key research frontiers: scalability, generalization, robustness, and human-AI alignment.

In reinforcement learning, reward function design faces critical challenges including delayed signals, ambiguity, misalignment with task objectives, and induction of undesirable behaviors. To address these, this paper proposes three novel reward mechanisms: teacher-driven, adaptive explainable, and agent-autonomous reward generation. Our core contributions are the first-ever adaptive explainable reward design method and a meta-learning–driven autonomous reward generation framework—enabling a paradigm shift from expert-guided reward specification to online inverse reward modeling by the agent. Technically, we integrate reward shaping, eXplainable AI (XAI)-informed reward modeling, policy-value alignment, and online inverse reward design. Experiments across multiple sparse-reward benchmarks demonstrate over 40% faster training convergence, significantly improved policy robustness, and high reward interpretability—validated by domain experts with 92% inter-rater agreement.

In reinforcement learning (RL), intrinsic motivation and reward shaping are prone to reward hacking—e.g., excessive fixation on stochastic noise sources like a flickering TV. Method: We unify these mechanisms as reward shaping within a Bayesian Adaptive Markov Decision Process (BAMDP) framework, introducing the BAMDP Potential Function (BAMPF). BAMPF decomposes state value into information-value and prior physical-state value components to guide robust exploration, and establishes boundedness criteria ensuring alignment between pseudo-rewards and true value. Contribution/Results: We provide the first theoretical proof that BAMPF is provably immune to reward hacking—within both meta-RL and standard RL settings. Empirically, BAMPF discovers optimal RL algorithms in Bernoulli Bandit meta-learning tasks and demonstrates effectiveness and plug-and-play compatibility in the Mountain Car environment.

In reinforcement learning, Linear Temporal Logic (LTL) tasks suffer from sparse rewards, hindering subgoal guidance, slowing policy convergence, and degrading robustness. To address this, we propose a progress-aware adaptive reward shaping method: for the first time, we quantify LTL satisfaction progress as a continuous, differentiable reward signal and design an online mechanism to dynamically update the reward function according to the agent’s current learning state. Our approach integrates LTL task compilation, formal progress modeling, and deep RL frameworks (PPO/SAC). Evaluated across multiple benchmark environments, the method significantly accelerates convergence, increases average expected return by 23%, improves task completion rate by 31%, and outperforms both traditional sparse-reward baselines and handcrafted reward-shaping approaches in terms of robustness and generalization.

To address low sample efficiency and unstable convergence in reinforcement learning caused by sparse rewards, this paper proposes an adaptive reward shaping method grounded in historical success rates. The method models state-dependent success probability as a time-varying Beta distribution—explicitly capturing epistemic uncertainty for the first time in this context. It further introduces an uncertainty-driven stochastic annealing strategy that naturally balances exploration and exploitation. For scalable, model-free, nonparametric success-rate estimation in high-dimensional continuous state spaces, the approach integrates kernel density estimation (KDE) with Random Fourier Features. Experiments demonstrate substantial improvements in sample efficiency and convergence stability on extremely sparse-reward tasks, consistently outperforming state-of-the-art reward shaping and intrinsic motivation baselines across diverse benchmarks.

For implicit-standard tasks—such as autonomous driving, humanoid locomotion, and dexterous manipulation—where “desirable behavior” is difficult to formalize and reward design suffers from subjectivity, this paper proposes REvolve: the first end-to-end reward evolution framework. REvolve integrates large language model (LLM)-driven reasoning, structured human feedback modeling, evolutionary algorithms, and deep reinforcement learning to automatically transform qualitative judgments into quantitative reward functions. Through iterative closed-loop cycles—comprising reward generation, pairwise ranking, mutation, and policy evaluation—it dynamically incorporates expert implicit knowledge into reward optimization. Evaluated on three canonical tasks, REvolve-generated reward functions yield policies with significantly improved performance, consistently outperforming state-of-the-art baselines including rule-based, inverse reinforcement learning, and zero-shot LLM-derived rewards.

This work addresses the challenge in reinforcement learning that handcrafted reward functions are time-consuming to design and often misaligned with task objectives. To this end, the paper introduces the Trajectory Alignment Coefficient (TAC) as a metric quantifying the consistency between a reward function and expert preferences, and for the first time employs TAC both as an auxiliary signal for reward tuning and as a direct learning objective for reward modeling. Furthermore, the authors develop Soft-TAC, a differentiable approximation of TAC, enabling end-to-end learning of reward models from human preference data. Experiments demonstrate that in Lunar Lander, TAC significantly improves reward function performance while reducing the cognitive burden of manual tuning; in Gran Turismo 7, reward models trained with Soft-TAC yield more diverse and effective behavioral policies compared to standard cross-entropy approaches.

Reward function design in reinforcement learning heavily relies on human expertise, resulting in poor generalizability and high engineering costs. Method: We propose the first fully autonomous framework for reward function generation and optimization—requiring no predefined evaluation metrics, environment source code, or human feedback. Leveraging large language models (LLMs), it integrates task-semantic parsing with multi-round sampling to enable model-agnostic, unsupervised generation, execution, and evaluation of reward functions. The LLM autonomously infers task-specific performance metrics and selects high-performing reward functions. Contribution/Results: Experiments across multiple control benchmarks demonstrate that our approach matches or surpasses state-of-the-art methods (e.g., EUREKA). Notably, it achieves competitive performance even with low-cost LLMs, substantially reducing human intervention while improving generality and automation in reward design.

This work addresses the challenge of sparse rewards in reinforcement learning, which hinders effective exploration, and the risk of reward gaming associated with handcrafted reward shaping. The authors propose the first integration of vision-language models (VLMs) into potential-based reward shaping (PBRS), leveraging a lightweight VLM to automatically learn a potential function by evaluating preferences over pairs of state images. This approach preserves the original optimal policy while eliminating human-induced design bias. Notably, the method requires only a small-scale VLM to efficiently generate preference labels, substantially improving sample efficiency. Empirical results in Meta-World and Franka Kitchen environments demonstrate strong robustness against reward gaming, confirming that even low-accuracy VLMs can effectively accelerate learning.

In multi-objective reinforcement learning (MORL), manually designed reward functions suffer from subjectivity and poor generalizability. To address this, we propose MORSE—a framework that automatically synthesizes multiple human-specified heuristic rewards into a unified, differentiable, and optimizable composite reward function via a bilevel optimization mechanism. Crucially, MORSE introduces exploratory noise derived from both task performance and prediction errors of a random neural network, enhancing policy exploration and mitigating local optima. Integrated with policy gradient optimization, MORSE is evaluated across diverse robot control tasks in MuJoCo and Isaac Sim. Results demonstrate that it achieves or surpasses the performance of hand-tuned reward functions in both Pareto optimality and overall task performance, while significantly reducing reward engineering effort.

This work addresses the challenge of suboptimal coordination in cooperative multi-agent reinforcement learning caused by misaligned handcrafted auxiliary rewards, particularly under sparse reward conditions. To overcome this, the authors propose an automated reward design framework that leverages large language models to generate executable reward programs from environmental observations, while incorporating formal validity constraints to ensure incentive alignment. Within a fixed computational budget, the framework selects the optimal reward function solely based on sparse task returns by training policies from scratch. The approach substantially reduces manual engineering effort and consistently improves both task return and delivery efficiency across four Overcooked-AI layouts, notably enhancing inter-agent action dependencies and alignment of coordination signals in scenarios with interaction bottlenecks.