OGBench: Benchmarking Offline Goal-Conditioned RL

2024-10-26

arXiv.org

AI Summary

Offline goal-conditioned reinforcement learning (GCRL) lacks standardized benchmarks, hindering systematic evaluation of algorithmic capabilities. To address this, we introduce OGBench—the first high-quality, capability-decoupled offline GCRL benchmark. It comprises eight environment categories and 85 diverse offline datasets, integrating both real-world and synthetic scenarios. We also open-source unified implementations of six representative algorithms, including behavior cloning, Q-learning variants, and inverse dynamics modeling. Crucially, OGBench is the first to orthogonally evaluate algorithms along four capability dimensions: behavior stitching, long-horizon reasoning, robustness to high-dimensional inputs, and handling of stochasticity. Empirical results reveal significant performance divergence across these dimensions, exposing critical limitations of existing methods. By providing a reproducible, quantifiable evaluation platform, OGBench fills a key gap in offline GCRL assessment and enables rigorous algorithm diagnosis, attribution analysis, and principled development of next-generation methods.

Technology Category

• Machine Learning: Online Learning & Bandits
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Offline goal-conditioned reinforcement learning (GCRL) is a major problem in reinforcement learning (RL) because it provides a simple, unsupervised, and domain-agnostic way to acquire diverse behaviors and representations from unlabeled data without rewards. Despite the importance of this setting, we lack a standard benchmark that can systematically evaluate the capabilities of offline GCRL algorithms. In this work, we propose OGBench, a new, high-quality benchmark for algorithms research in offline goal-conditioned RL. OGBench consists of 8 types of environments, 85 datasets, and reference implementations of 6 representative offline GCRL algorithms. We have designed these challenging and realistic environments and datasets to directly probe different capabilities of algorithms, such as stitching, long-horizon reasoning, and the ability to handle high-dimensional inputs and stochasticity. While representative algorithms may rank similarly on prior benchmarks, our experiments reveal stark strengths and weaknesses in these different capabilities, providing a strong foundation for building new algorithms. Project page: https://seohong.me/projects/ogbench

Problem

• Develops benchmark for offline goal-conditioned RL
• Evaluates algorithm capabilities systematically
• Includes diverse environments and datasets

Innovation

• Introduces OGBench benchmark
• Includes 8 environments, 85 datasets
• Evaluates 6 offline GCRL algorithms