UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning

2026-10-02

AI Summary

This study addresses the mismatch between fixed intervention policies and dynamic robotic capabilities in online reinforcement learning by proposing an adaptive intervention agent based on semi-Markov decision processes. Methodologically, it introduces a novel capability-aware periodic probing mechanism coupled with an experience learning framework. Through online value estimation, this approach jointly optimizes intervention timing, modality, and control handover, thereby achieving a dynamic balance between autonomous execution and human assistance. Evaluated across five real-world manipulation tasks, the proposed method attains an average success rate of 89.67% while reducing the human intervention rate to 0.77%, significantly outperforming existing baselines.

Abstract

Online reinforcement learning (RL) enables robot policies to improve through physical interaction, but the assistance they require changes as their competence evolves. Existing intervention strategies based on offline estimates or fixed decision rules can therefore become mismatched to the current policy. To address this, we propose UniIntervene++, an adaptive intervention agent that learns to allocate control between autonomous execution and heterogeneous assisted behaviors during online RL. Specifically, UniIntervene++ first formulates the evolving RL policy, trajectory correction, and a task-structured CodePolicy as Options in a unified semi-Markov decision process and learns their relative values online. Building on this, competence-adaptive intervention periodically probes the RL policy through unassisted execution, keeping control allocation responsive to its evolving capability. Finally, coupled experience learning allows assisted behaviors to improve the RL policy, whose evolving outcomes in turn reshape future intervention decisions. In this way, UniIntervene++ jointly determines when to intervene, how to intervene, and when to return control as the RL policy improves. Across five real-world manipulation tasks, UniIntervene++ achieves an average success rate of 89.67%, outperforming all baselines by at least 6 percentage points, while reducing human intervention to 0.77%, a relative reduction of at least 94.6% from the best baseline. Code is available in our \href{https://github.com/dannyyudong/An-Adaptive-Intervention-Agent-for-Efficient-Real-World-Reinforcement-Learning}{GitHub repository}.

Problem

• Reinforcement Learning
• Intervention Strategy
• Robot Manipulation
• Online Learning
• Human Intervention

Innovation

• Adaptive Intervention
• Reinforcement Learning
• Semi-Markov Decision Process
• Coupled Experience Learning
• Real-World Robotics