Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback

2025-07-17

AI Summary

Reinforcement learning (RL) suffers from inefficient policy learning under sparse rewards, while existing RL from human feedback (RLHF) methods rely on explicit, effortful user inputs that disrupt natural interaction and impose high cognitive load. Method: We propose the first human-robot behavioral alignment framework grounded in implicit neurofeedback—specifically, error-related potentials (ErrPs)—decoded in real time via non-invasive electroencephalography (EEG). This yields a probabilistic reward signal without requiring active user responses. Integrating a pre-trained EEG decoder with RL, we train a Kinova Gen2 robotic arm in MuJoCo to perform obstacle-avoiding grasping. Contribution/Results: Our approach matches the performance of dense manual reward baselines while substantially reducing user cognitive burden. It represents the first end-to-end robot learning system for continuous control driven solely by implicit neural feedback, advancing human-robot collaboration toward greater naturalness and scalability.

Technology Category

Intelligent Robots: Human-Robot InteractionHumans and AI: Human-Aware Planning and Behavior PredictionMachine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

Responsible Web: Machine-in-the-loop, human agency and autonomyEconomics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applicationsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Conventional reinforcement learning (RL) ap proaches often struggle to learn effective policies under sparse reward conditions, necessitating the manual design of complex, task-specific reward functions. To address this limitation, rein forcement learning from human feedback (RLHF) has emerged as a promising strategy that complements hand-crafted rewards with human-derived evaluation signals. However, most existing RLHF methods depend on explicit feedback mechanisms such as button presses or preference labels, which disrupt the natural interaction process and impose a substantial cognitive load on the user. We propose a novel reinforcement learning from implicit human feedback (RLIHF) framework that utilizes non-invasive electroencephalography (EEG) signals, specifically error-related potentials (ErrPs), to provide continuous, implicit feedback without requiring explicit user intervention. The proposed method adopts a pre-trained decoder to transform raw EEG signals into probabilistic reward components, en abling effective policy learning even in the presence of sparse external rewards. We evaluate our approach in a simulation environment built on the MuJoCo physics engine, using a Kinova Gen2 robotic arm to perform a complex pick-and-place task that requires avoiding obstacles while manipulating target objects. The results show that agents trained with decoded EEG feedback achieve performance comparable to those trained with dense, manually designed rewards. These findings validate the potential of using implicit neural feedback for scalable and human-aligned reinforcement learning in interactive robotics.

Problem

• Overcoming sparse rewards in RL without manual reward design
• Reducing cognitive load by replacing explicit human feedback
• Using EEG signals for continuous implicit feedback in RL

Innovation

• Utilizes EEG signals for implicit feedback
• Transforms raw EEG into probabilistic rewards
• Enables policy learning with sparse rewards