🤖 AI Summary
This work addresses the challenge of suboptimal coordination in cooperative multi-agent reinforcement learning caused by misaligned handcrafted auxiliary rewards, particularly under sparse reward conditions. To overcome this, the authors propose an automated reward design framework that leverages large language models to generate executable reward programs from environmental observations, while incorporating formal validity constraints to ensure incentive alignment. Within a fixed computational budget, the framework selects the optimal reward function solely based on sparse task returns by training policies from scratch. The approach substantially reduces manual engineering effort and consistently improves both task return and delivery efficiency across four Overcooked-AI layouts, notably enhancing inter-agent action dependencies and alignment of coordination signals in scenarios with interaction bottlenecks.
📝 Abstract
Designing effective auxiliary rewards for cooperative multi-agent systems remains a precarious task; misaligned incentives risk inducing suboptimal coordination, especially where sparse task feedback fails to provide sufficient grounding. This study introduces an automated reward design framework that leverages large language models to synthesize executable reward programs from environment instrumentation. The procedure constrains candidate programs within a formal validity envelope and evaluates their efficacy by training policies from scratch under a fixed computational budget; selection depends exclusively on the sparse task return. The framework is evaluated across four distinct Overcooked-AI layouts characterized by varied corridor congestion, handoff dependencies, and structural asymmetries. Iterative search generations consistently yield superior task returns and delivery counts, with the most pronounced gains occurring in environments dominated by interaction bottlenecks. Diagnostic analysis of the synthesized shaping components indicates increased interdependence in action selection and improved signal alignment in coordination-intensive tasks. These results demonstrate that the search for objectivegrounded reward programs can mitigate the burden of manual engineering while producing shaping signals compatible with cooperative learning under finite budgets.