GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

2026-09-27

AI Summary

This work addresses the susceptibility of divide-and-conquer methods to lucky trajectory bias under stochastic dynamics and the absence of value updates for disconnected state-goal pairs in offline goal-conditioned reinforcement learning. We propose GTRL, a novel algorithm that pioneers embedding one-step TD targets as base cases within divide-and-conquer composition rules rather than merely substituting them, thereby ensuring all state-goal pairs receive updates. Additionally, we introduce reachability weighting to correct estimation biases arising from hindsight relabeling while preserving long-horizon planning capabilities. By integrating techniques from offline reinforcement learning, temporal difference learning, and importance sampling, our method achieves the highest average success rate across nineteen tasks on the OGBench benchmark, encompassing stochastic, deterministic, and stitching environments.

Abstract

In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal. However, under stochastic dynamics, the base case of this rule values the luckiest trajectories through the data. The subgoal must also lie on a shared trajectory, so a state-goal pair that no trajectory connects gets no value update at all. To address both, we present Grounded Transitive RL (GTRL), an offline GCRL value learning algorithm that grounds the divide-and-conquer update with a one-step TD target. Over a single step, TD is correct, as its target averages over the successors and needs no subgoal. GTRL adds this target to the composition rather than replacing it, so every pair receives an update, and the composition still carries the long horizon. GTRL also corrects the bias from hindsight relabeling by reweighting each goal against how reachable it was from other successors. We evaluate our algorithm on nineteen OGBench tasks spanning stochastic, deterministic, and stitching environments, where it achieves the highest average success rate. Code will be released soon.

Problem

• offline goal-conditioned reinforcement learning
• divide-and-conquer
• stochastic dynamics
• value learning
• hindsight relabeling

Innovation

• offline goal-conditioned reinforcement learning
• divide-and-conquer value learning
• temporal difference
• hindsight relabeling
• subgoal composition