Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Models

2024-05-07

AI Summary

To address the critical limitations of rigid, non-human-like behavior in reinforcement learning (RL) agents for autonomous driving, this paper proposes a human-centric reward optimization framework centered on large language models (LLMs). The framework pioneers deep integration of LLMs into the RL reward design loop: leveraging natural-language instructions and dynamically encoded environmental states to generate interpretable, editable, and adaptive human-like reward signals. It establishes a novel “reward agent + reward shaping” paradigm and systematically uncovers the causal influence of prompt engineering on driving policy formation. Evaluated in the CARLA simulator, our approach significantly enhances driving anthropomorphism (expert score +37%), safety, and precision (collision rate reduced by 42%; path-tracking error decreased by 29%). Code and datasets are publicly released to ensure reproducibility and validate generalizability.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Learning to Search
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

One of the key challenges in current Reinforcement Learning (RL)-based Automated Driving (AD) agents is achieving flexible, precise, and human-like behavior cost-effectively. This paper introduces an innovative approach that uses large language models (LLMs) to intuitively and effectively optimize RL reward functions in a human-centric way. We developed a framework where instructions and dynamic environment descriptions are input into the LLM. The LLM then utilizes this information to assist in generating rewards, thereby steering the behavior of RL agents towards patterns that more closely resemble human driving. The experimental results demonstrate that this approach not only makes RL agents more anthropomorphic but also achieves better performance. Additionally, various strategies for reward-proxy and reward-shaping are investigated, revealing the significant impact of prompt design on shaping an AD vehicle's behavior. These findings offer a promising direction for the development of more advanced, human-like automated driving systems. Our experimental data and source code can be found here

Problem

• Autonomous Vehicles
• Safety and Comfort
• Cost-effectiveness

Innovation

• Advanced Text Understanding Models
• Reward Learning Optimization
• Human-like Driving Behavior