GORDON: Graph-based Object-centric Rewards for Decomposition of Long-Horizon Manipulation

2026-08-04

AI Summary

This work addresses the challenges of sparse rewards, intricate reward engineering, and costly manual subtask annotations in long-horizon robotic manipulation tasks. The authors propose a graph-based, object-centric reward learning framework that enables self-supervised dense reward acquisition from action-free video demonstrations alone. By modeling scenes as graphs of objects and their spatial relationships, the method employs graph neural networks to embed these structures into a task-aligned latent space, augmented with an activity-aware weighted pooling mechanism to emphasize task-relevant objects. Task progress is measured via the distance between the current state and the goal configuration in this latent space, and subtask phases are automatically discovered to train a hierarchical policy. Evaluated on seven tasks from MAGICAL and ManiSkill3, the approach achieves an average success rate of 74.4% without any human annotation, outperforming the best learning baseline and an oracle baseline by approximately 35 and 25 percentage points, respectively.

Abstract

Learning long-horizon manipulation skills with reinforcement learning remains challenging due to the complexity of reward design, the limited guidance of sparse rewards, and the high cost of manual subtask annotation. Visual demonstrations can provide supervision for reward learning, but rewards learned from raw pixels can be brittle and sensitive to visual variation, background appearance, and robot motion. In this work, we propose GORDON, a graph-based object-centric reward learning framework that learns dense rewards from action-free video demonstrations. Each visual scene is represented as a graph of detected objects and spatial relations, and a graph neural network is trained in a self-supervised manner to embed these graphs into a task-aligned latent space. To align the representation with semantic task progress, we introduce an activity-aware weighted pooling mechanism that emphasizes task-relevant objects while masking robot-dominated motion. The dense reward is then computed as distances in the learned latent space of the current state to demonstrated goal configurations, providing a measure of task progress. In long-horizon tasks, the temporal profile of this reward reveals stage-wise object-state transitions, enabling automatic subtask discovery without manual segmentation. The discovered segments are then used to train subtask-specific rewards and specialized policies that are composed sequentially. Experiments on seven manipulation tasks on MAGICAL and ManiSkill3 benchmarks show that our object-centric reward improves reinforcement learning in short-horizon settings and enables successful policy learning in complex long-horizon tasks through automatic decomposition, achieving an average success rate of 74.4% across the long-horizon tasks (on average approximately +35 p.p. vs. best learned baseline and approximately +25 p.p. vs. oracle).

Problem

• long-horizon manipulation
• reward design
• visual demonstrations
• subtask decomposition
• reinforcement learning

Innovation

• graph-based reward learning
• object-centric representation
• self-supervised embedding
• automatic subtask discovery
• long-horizon manipulation