Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?

2026-09-30

AI Summary

This study investigates how goal representation quality influences the performance of offline goal-conditioned reinforcement learning. While existing research predominantly focuses on optimizing goal representations, their practical benefits remain unclear. To address this, we systematically evaluate the impact of goal and state representations on downstream tasks within deterministic maze environments by constructing precise temporal-distance representations and introducing geometric perturbation noise. Our findings challenge conventional assumptions, revealing that the primary bottleneck in navigation tasks lies in current state representations rather than goal representations. Motivated by this insight, we propose a simple yet effective strategy employing random Fourier positional encoding to enhance state representations. This approach yields substantial improvements in execution success rates on challenging tasks within the OGBench benchmark.

Abstract

Goal-conditioned reinforcement learning (GCRL) relies heavily on how target goals are represented to the policy. While recent methods encode goals via temporal distance, occupancy, or controllability, it remains unclear how much downstream performance actually depends on representation quality. We study this in offline GCRL by constructing an exact temporal-distance goal representation in deterministic mazes. We then systematically corrupt its geometric quality while keeping the downstream learner fixed. Across OGBench navigation tasks and two algorithms, large changes in goal-representation quality produce almost no change in performance. However, applying the same interventions to the agent's current state more than doubles success, revealing the state pathway as the true bottleneck. Building on this insight, we show that simple random Fourier positional encodings substantially improve performance on the hardest navigation tasks without map information or objective modifications. Overall, our findings suggest that in state-based offline navigation, improving how the agent's current state is represented matters far more than refining the goal representation. Code will be released soon.

Problem

• Goal-Conditioned Reinforcement Learning
• Goal Representation
• Offline RL
• State Representation

Innovation

• Goal-Conditioned Reinforcement Learning
• State Representation
• Random Fourier Positional Encodings
• Offline RL
• Navigation