Goal Sets, Not Goal States: Queryable Robot Goals through Goal-Set Hindsight Relabeling

2026-06-08

AI Summary

This work addresses a key limitation in conventional hindsight experience replay (HER), which treats achieved states as precise goal targets and thereby imposes overly restrictive constraints when tasks depend only on subsets of the state space, ultimately hindering performance in offline goal-conditioned reinforcement learning. To overcome this, the authors propose Goal-Set Hindsight Experience Replay (GS-HER), which maps achieved states to goal sets defined by binary predicates rather than individual goal states. This formulation enables dynamic support for arbitrary goal predicates during inference and generalizes relabeling to the predicate level, allowing a single policy to generalize across diverse goal semantics without retraining. Experiments on OGBench with five offline GCRL algorithms demonstrate that GS-HER substantially improves performance, particularly when the full state observation contains irrelevant dimensions.

Abstract

Hindsight relabeling usually turns achieved future states into exact goals, which can overconstrain offline robot learning when task success depends only on a subset of the state. We propose Goal-Set Hindsight Relabeling (GS-HER), a predicate-level generalization of HER in which achieved states certify query-defined goal sets rather than singleton goal states. A binary query specifies which variables define success, making the goal predicate an inference-time input while leaving the underlying offline GCRL algorithm unchanged. Across OGBench tasks and five offline goal-conditioned learners, GS-HER improves performance when full-state goals are bottlenecked by nuisance dimensions and turns hindsight relabeling into a reusable goal interface: one checkpoint can answer multiple robot goal predicates without retraining.

Problem

• hindsight relabeling
• goal-conditioned reinforcement learning
• offline robot learning
• goal sets
• nuisance dimensions

Innovation

• Goal-Set Hindsight Relabeling
• goal-conditioned reinforcement learning
• offline robot learning
• goal predicates
• reusable goal interface