Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition

2026-08-10

AI Summary

This work addresses key challenges in online reinforcement learning for robotic manipulation—namely low sample efficiency, non-stationarity in multi-agent settings, and limited effectiveness of domain randomization—by proposing a unified framework that integrates centralized training with decentralized execution (CTDE) and a hybrid reward architecture (HRA). The approach decomposes the centralized critic into semantically distinct task and grasp heads, and redesigns the actor-critic objective by incorporating potential-based grasp rewards and categorical action distribution modeling. This formulation effectively mitigates sparse reward signals and enables robust performance under extensive domain randomization. Empirical results demonstrate substantial improvements in task success rates on both real dual-arm robots and simulated humanoid platforms: tennis ball pick-and-place performance rises from 60% to 80%, banana pick-and-place achieves 90%, simulated block relocation jumps from 25% to 95%, and tasks completely failed by baseline methods are successfully accomplished.

Abstract

Real-world online reinforcement learning (RL) provides a promising approach for training robotic manipulation policies directly in the physical world, avoiding the sim-to-real gap and enabling continuous policy refinement through human-in-the-loop interaction. Recent methods have demonstrated sample-efficient learning through human intervention but remain limited to small randomization ranges and encounter challenges with the non-stationarity induced by concurrently training multiple agents. To address these limitations, we introduce a unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA). This enables multiple actors to share a centralized multi-head critic. The critic is decomposed into task and grasp heads, corresponding to the sparse task reward and a potential-based grasping reward, respectively. We accordingly reformulate the critic and actor objectives to exploit the decomposed Q-values while explicitly accounting for the categorical action distribution of the discrete gripper policy. Experimental results demonstrate that the proposed framework substantially improves both sample efficiency and policy performance. We validate our approach on two robotic arms and a simulated humanoid robot across tennis ball and banana pick-and-place, pot reset, and simulated block relocation tasks under dimension-wise domain randomization, approximately 5-25x larger than those considered in prior work. Compared with a state-of-the-art baseline, our method improves the success rate from 60% to 80% on tennis ball pick-and-place, from 60% to 90% on banana pick-and-place, and from 25% to 95% on simulated block relocation, while also successfully accomplishing a task where the baseline consistently fails. Videos and more details are available at our project website: https://hil-harc.github.io/.

Problem

• real-world online reinforcement learning
• robot manipulation
• non-stationarity
• domain randomization
• multi-agent training

Innovation

• Centralized Training with Decentralized Execution
• Critic Decomposition
• Hybrid Reward Architecture
• Online Reinforcement Learning
• Domain Randomization