TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decomposition

2026-06-03

AI Summary

Reinforcement learning struggles in zero-reward settings due to the absence of optimization signals, making it difficult to learn from entirely unsuccessful trajectories. This work proposes TD-Grokking, a training-time decomposition framework that introduces, for the first time, a recursive problem decomposition mechanism. By recursively breaking down complex root problems into verifiable subproblems and organizing them into a hierarchical tree structure, TD-Grokking enables leaf nodes to generate non-zero reward signals that drive learning. This approach overcomes the limitations of traditional reinforcement learning from very rare rewards (RLVR) in environments devoid of explicit rewards. Experimental results demonstrate that TD-Grokking significantly outperforms the original GRPO algorithm and existing baselines on mathematical and medical reasoning tasks, effectively generating usable training signals and achieving consistent performance gains.

Abstract

Large language models (LLMs) have made remarkable progress in reasoning tasks, largely driven by post-training paradigms, especially reinforcement learning with verifiable rewards (RLVR). However, a critical bottleneck persists: RLVR fails on highly challenging zero-reward problems, where all sampled reasoning trajectories yield uniformly failed outcomes, providing no optimization signal to drive model improvement. Prior efforts to address this limitation, such as dense process supervision, partial reward assignment, or prefix-guided exploration, suffer from inherent task constraints or do not fully equip the policy model with the capabilities necessary to solve the original intractable problems. To address this, we propose TD-Grokking, a training-time decomposition framework for zero-reward problems. It recursively decomposes intractable root problems into self-contained, verifiable subproblems, forming hierarchical trees where solvable leaves provide non-zero rewards. Evaluations on mathematical and medical tasks show that TD-Grokking outperforms vanilla GRPO as well as all baseline approaches. Together with detailed analysis, these results confirm that training-time decomposition effectively converts zero-reward examples into usable training signals, enabling consistent performance gains. Our code and datasets are available at https://anonymous.4open.science/r/TD-Grokking-6567/.

Problem

• zero-reward problems
• reinforcement learning
• reasoning tasks
• optimization signal
• intractable problems

Innovation

• training-time decomposition
• zero-reward problems
• hierarchical problem solving
• reinforcement learning with verifiable rewards
• LLM reasoning