🤖 AI Summary
Reward function design in reinforcement learning heavily relies on human expertise, resulting in poor generalizability and high engineering costs.
Method: We propose the first fully autonomous framework for reward function generation and optimization—requiring no predefined evaluation metrics, environment source code, or human feedback. Leveraging large language models (LLMs), it integrates task-semantic parsing with multi-round sampling to enable model-agnostic, unsupervised generation, execution, and evaluation of reward functions. The LLM autonomously infers task-specific performance metrics and selects high-performing reward functions.
Contribution/Results: Experiments across multiple control benchmarks demonstrate that our approach matches or surpasses state-of-the-art methods (e.g., EUREKA). Notably, it achieves competitive performance even with low-cost LLMs, substantially reducing human intervention while improving generality and automation in reward design.
📝 Abstract
The challenge of designing effective reward functions in reinforcement learning (RL) represents a significant bottleneck, often requiring extensive human expertise and being time-consuming. Previous work and recent advancements in large language models (LLMs) have demonstrated their potential for automating the generation of reward functions. However, existing methodologies often require preliminary evaluation metrics, human-engineered feedback for the refinement process, or the use of environmental source code as context. To address these limitations, this paper introduces LEARN-Opt (LLM-based Evaluator and Analyzer for Reward functioN Optimization). This LLM-based, fully autonomous, and model-agnostic framework eliminates the need for preliminary metrics and environmental source code as context to generate, execute, and evaluate reward function candidates from textual descriptions of systems and task objectives. LEARN-Opt's main contribution lies in its ability to autonomously derive performance metrics directly from the system description and the task objective, enabling unsupervised evaluation and selection of reward functions. Our experiments indicate that LEARN-Opt achieves performance comparable to or better to that of state-of-the-art methods, such as EUREKA, while requiring less prior knowledge. We find that automated reward design is a high-variance problem, where the average-case candidate fails, requiring a multi-run approach to find the best candidates. Finally, we show that LEARN-Opt can unlock the potential of low-cost LLMs to find high-performing candidates that are comparable to, or even better than, those of larger models. This demonstrated performance affirms its potential to generate high-quality reward functions without requiring any preliminary human-defined metrics, thereby reducing engineering overhead and enhancing generalizability.