design reward functions

This paper addresses the misalignment between reward models and true objectives in deep reinforcement learning, as well as the resulting limitations in policy optimization. To this end, it introduces— for the first time—a unified taxonomy that systematically organizes reward modeling across three orthogonal dimensions: modeling source (explicit vs. implicit), mechanism design (supervised vs. interactive), and learning paradigm (static vs. dynamic). The survey comprehensively covers mainstream approaches—including inverse reinforcement learning, preference learning, language-model-based feedback, human demonstration distillation, contrastive learning, and online interactive modeling—and critically analyzes evaluation methodologies and practical deployment challenges. This work fills a critical gap in the literature by providing the first systematic, cross-cutting review of reward modeling. It clarifies the technical evolution of the field and identifies four key research frontiers: scalability, generalization, robustness, and human-AI alignment.

Reward function design in reinforcement learning heavily relies on human expertise, resulting in poor generalizability and high engineering costs. Method: We propose the first fully autonomous framework for reward function generation and optimization—requiring no predefined evaluation metrics, environment source code, or human feedback. Leveraging large language models (LLMs), it integrates task-semantic parsing with multi-round sampling to enable model-agnostic, unsupervised generation, execution, and evaluation of reward functions. The LLM autonomously infers task-specific performance metrics and selects high-performing reward functions. Contribution/Results: Experiments across multiple control benchmarks demonstrate that our approach matches or surpasses state-of-the-art methods (e.g., EUREKA). Notably, it achieves competitive performance even with low-cost LLMs, substantially reducing human intervention while improving generality and automation in reward design.

This work addresses key challenges in multi-agent reinforcement learning—such as ambiguous credit assignment, environmental non-stationarity, and complex agent interactions—stemming from handcrafted reward functions. To overcome these limitations, the paper proposes leveraging large language models (LLMs) to directly translate natural language objectives into semantic reward signals, replacing conventional hand-designed numerical rewards. The approach restructures coordination mechanisms through three pillars: semantic reward specification, dynamic adaptation, and alignment with human intent, enabling agents to collaborate based on shared semantic representations rather than explicit numeric cues. By integrating LLMs (e.g., EUREKA, CARD) with the Verifiable Reward Reinforcement Learning (RLVR) framework, the method achieves language-driven reward generation and online optimization. Experimental results demonstrate that this paradigm significantly reduces manual intervention while substantially improving alignment between multi-agent behavior and human intentions.

In reinforcement learning, Linear Temporal Logic (LTL) tasks suffer from sparse rewards, hindering subgoal guidance, slowing policy convergence, and degrading robustness. To address this, we propose a progress-aware adaptive reward shaping method: for the first time, we quantify LTL satisfaction progress as a continuous, differentiable reward signal and design an online mechanism to dynamically update the reward function according to the agent’s current learning state. Our approach integrates LTL task compilation, formal progress modeling, and deep RL frameworks (PPO/SAC). Evaluated across multiple benchmark environments, the method significantly accelerates convergence, increases average expected return by 23%, improves task completion rate by 31%, and outperforms both traditional sparse-reward baselines and handcrafted reward-shaping approaches in terms of robustness and generalization.

Current large language models (LLMs) rely on supervised fine-tuning (SFT) for tool-use learning, exhibiting poor generalization; reinforcement learning (RL) approaches are hindered by coarse-grained rewards (e.g., final answer matching), failing to guide fine-grained tool selection and parameter invocation. Method: We propose the first multi-dimensional reward design framework tailored for tool-calling tasks, systematically characterizing reward types, granularity, and temporal structure to establish a principled, fine-grained reward mechanism. Leveraging Group Relative Policy Optimization (GRPO), we enable end-to-end RL training for tool calling. Contribution/Results: Our method achieves significant improvements—+15% over SFT baselines and +17% absolute gain across multiple benchmarks—while demonstrating enhanced training robustness, scalability, and stability.

In reinforcement learning, reward modeling suffers from “error-regret mismatch”: low test error of the reward model does not guarantee low regret of the optimized policy, primarily due to distributional shift induced by policy optimization. Method: We provide the first theoretical proof that, for any arbitrarily small expected test error, there exist underlying data distributions yielding arbitrarily large regret. We construct explicit counterexamples, derive tight quantitative bounds linking reward estimation error and policy regret, and analyze the robustness of regularization techniques—including RLHF—against this mismatch. Contribution/Results: We show that low test error only ensures a worst-case regret upper bound, not actual policy performance; moreover, standard regularizers fail to eliminate the mismatch. Our analysis establishes a new theoretical benchmark for assessing reward model reliability and safety alignment in preference-based RL, with implications for trustworthy reward learning and deployment-critical applications.

This work addresses the limitations of large language models in generating high-quality BPMN process models, which are constrained by supervised fine-tuning data and the absence of well-defined multidimensional reward functions. The authors propose a reinforcement learning–based optimization approach that systematically explores a reward function encompassing 38 syntactic, pragmatic, and semantic metrics. They train Llama-3.1-8B and Qwen2.5-14B models across 48 configurations and find that uniformly weighted rewards outperform targeted weighting schemes, with significant interaction effects observed between reward composition and model architecture. Leveraging Group Relative Policy Optimization and an automated evaluation framework, the method substantially improves pragmatic and syntactic quality while preserving semantic fidelity and reducing output variability by over sixfold. All code is publicly released.

Sparse, delayed, and weakly informative reward signals severely hinder the efficiency of reinforcement learning, and existing reward shaping methods often fail to adapt to dynamic environments. This work proposes the first unified analytical framework encompassing temporal, informational, and theoretical dimensions to systematically categorize and compare twelve classes of dynamic reward shaping and related adaptive mechanisms. It clearly distinguishes between parameter corrections and state-dependent modifications, and precisely delineates the boundaries among additive shaping, reward replacement, and correlated guidance. Through integrated theoretical analysis and taxonomic synthesis, the study identifies conditions under which optimality is preserved in the presence of modern RL components such as experience replay, bootstrapped critics, and reward normalization, and for the first time elucidates the intrinsic relationship between adaptation rate and learning stability.

This work addresses the challenge of sparse rewards in reinforcement learning, which hinders effective exploration, and the risk of reward gaming associated with handcrafted reward shaping. The authors propose the first integration of vision-language models (VLMs) into potential-based reward shaping (PBRS), leveraging a lightweight VLM to automatically learn a potential function by evaluating preferences over pairs of state images. This approach preserves the original optimal policy while eliminating human-induced design bias. Notably, the method requires only a small-scale VLM to efficiently generate preference labels, substantially improving sample efficiency. Empirical results in Meta-World and Franka Kitchen environments demonstrate strong robustness against reward gaming, confirming that even low-accuracy VLMs can effectively accelerate learning.