Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving

2025-04-28

AI Summary

To address the challenge of effectively integrating human expert intervention and evaluation into reinforcement learning for autonomous driving, this paper proposes an interactive human-in-the-loop double deep Q-network (ID-DQN) framework. Methodologically: (1) a human-in-the-loop (HITL) interface dynamically incorporates real-time human interventions into the DQN decision-making process; (2) an offline counterfactual evaluation module quantifies intervention efficacy to guide policy optimization; and (3) the Q-value update mechanism of Double DQN is refined to improve robustness and interpretability. Evaluated on simulated driving tasks, ID-DQN significantly outperforms behavioral cloning (BC), HG-DAgger, DQfD, and standard deep RL baselines—achieving a 23.6% improvement in policy success rate and markedly enhanced generalization to unseen scenarios. This work constitutes the first end-to-end co-modeling of human interventions and Q-learning, establishing a verifiable human–machine shared control paradigm for safety-critical autonomous driving.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Human-Robot Interaction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Integrating human expertise with machine learning is crucial for applications demanding high accuracy and safety, such as autonomous driving. This study introduces Interactive Double Deep Q-network (iDDQN), a Human-in-the-Loop (HITL) approach that enhances Reinforcement Learning (RL) by merging human insights directly into the RL training process, improving model performance. Our proposed iDDQN method modifies the Q-value update equation to integrate human and agent actions, establishing a collaborative approach for policy development. Additionally, we present an offline evaluative framework that simulates the agent's trajectory as if no human intervention had occurred, to assess the effectiveness of human interventions. Empirical results in simulated autonomous driving scenarios demonstrate that iDDQN outperforms established approaches, including Behavioral Cloning (BC), HG-DAgger, Deep Q-Learning from Demonstrations (DQfD), and vanilla DRL in leveraging human expertise for improving performance and adaptability.

Problem

• Enhance RL in autonomous driving with human insights integration
• Develop collaborative policy via modified Q-value update equation
• Evaluate human intervention effectiveness using offline simulation framework

Innovation

• Integrates human insights into RL training
• Modifies Q-value update for human-agent collaboration
• Offline evaluative framework assesses human intervention impact