ReF-HIL: Shaping the Critic around Human Action Neighborhoods for Efficient Human-in-the-Loop Reinforcement Learning

2026-09-29

AI Summary

This study addresses the inefficient training in human-in-the-loop reinforcement learning (HIL-RL) caused by underutilized human experience and imitation penalties that constrain policy optimization. To overcome these limitations, this work proposes the ReF-HIL framework, which innovatively integrates independent value reference learning with a dynamic human action neighborhood mechanism. Specifically, it introduces human-reference-guided value shaping to eliminate intrinsic imitation penalties and accelerate learning, while defining a human action fence to constrain extrinsic updates, thereby achieving an effective balance between autonomous exploration and human demonstration. Evaluated across five real-world robotic tasks, the proposed framework attains a 90% success rate within merely 18 to 63 minutes of training, ultimately reaching final success rates of 91.7% to 100%. These results demonstrate that ReF-HIL significantly enhances learning efficiency in HIL-RL settings.

Abstract

Human-in-the-loop reinforcement learning (HIL-RL) offers a promising route to efficient training of robotic manipulation policies by combining autonomous learning with human demonstrations and online corrections. However, insufficient use of successful human experience in value learning prolongs costly real-world training, while persistent imitation penalties can limit value-driven policy improvement. To address these limitations, we propose ReF-HIL, an efficient HIL-RL framework that uses human guidance to accelerate the learning process. Human-Reference-Guided Value Shaping learns an independent value reference from successful human experience to guide online value learning, while incorporating local corrective feedback. A Human Action Fence defines a learned human-action neighborhood, allowing value-driven optimization for better performance without imitation penalties inside while constraining policy and value updates outside. Experiments on five diverse and challenging real-world manipulation tasks demonstrate improved overall learning efficiency and higher success rates compared with the evaluated baselines. Specifically, ReF-HIL reaches 90% autonomous success in only 18-63 minutes of active training and achieves final success rates of 91.7-100%. These results highlight the potential of human-guided reinforcement learning to acquire reliable manipulation skills efficiently in the real world. Project website: https://anonymous.4open.science/w/ReF-HIL-7762/

Problem

• Human-in-the-loop reinforcement learning
• Robotic manipulation
• Value learning
• Imitation penalty
• Learning efficiency

Innovation

• Human-in-the-loop Reinforcement Learning
• Value Shaping
• Human Action Fence
• Robotic Manipulation
• Imitation Penalty