EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning

2026-08-04

AI Summary

This work addresses key challenges in human-in-the-loop reinforcement learning, including the static nature of visual reward models, temporal inconsistency in action sampling, and policy sensitivity to visual appearance. To overcome these issues, the authors propose a unified co-optimization framework that jointly trains the reward model, action generator, and visual representation through staged human-robot interaction. Core innovations include a self-evolving reward mechanism, flow-matching-based coherent action generation, and perception-preserving offline fine-tuning that integrates relighting replay with behavioral anchoring—enabling visual domain adaptation without additional human interaction. Evaluated on six contact-intensive manipulation tasks, the approach significantly improves task success rate, alignment with human preferences, action smoothness, and execution efficiency.

Abstract

Human-in-the-loop reinforcement learning (HIL-RL) enables robots to learn contact-rich manipulation from limited real-world interaction, but deployment exposes three coupled limitations: static visual reward models fail under scene changes; independently sampled actions cause temporally inconsistent motion; and vision-based policies remain sensitive to appearance shifts. We present EvoHIL, a unified framework that adapts the reward model, action generator, and visual do main within a staged human-in-the-loop learning process. First, self-evolving reward (SER) adapts the success classifier from human-confirmed positives and provisional weak negatives. Second, Action Flow Stabilization (AFS) generates temporally coherent action chunks through flow matching, grounding policy updates in executed action prefixes and demonstrated behavior. Third, retention-aware offline fine-tuning replays relit interaction data while anchoring the AFS actor-critic to prior behavior, adapting the visual domain without additional robot interaction. Across six manipulation tasks on Franka FR3 and SO-101 arms under a controlled lighting shift, EvoHIL improves task success, agreement with human-confirmation labels, motion smoothness, and completion time relative to human-in-the-loop and imitation baselines.Project page: https://anonymous4366.github.io/EvoHIL/

Problem

• Human-in-the-loop reinforcement learning
• visual reward model
• temporal action consistency
• appearance robustness
• contact-rich manipulation

Innovation

• Self-Evolving Reward
• Flow-Matched Policy Optimization
• Human-in-the-Loop Reinforcement Learning
• Action Flow Stabilization
• Retention-Aware Offline Fine-Tuning