BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

2026-08-03

AI Summary

This study investigates whether reinforcement learning with verifiable rewards (RLVR) genuinely enhances the reasoning capabilities of large language models or merely improves sampling efficiency. To this end, we introduce BODHI-Trees—a novel tree-based representation that extracts semantically equivalent structures from mathematical reasoning trajectories—and propose semantic branching entropy as a new metric to quantify reasoning diversity. Through controlled maze experiments and trajectory analyses, we find that while RLVR strengthens constraint adherence and backtracking abilities, it substantially contracts the semantic reasoning space, leading to a concurrent collapse in both policy entropy and semantic branching entropy. These findings suggest that the efficiency gains conferred by RLVR may come at the cost of reduced reasoning diversity.

Abstract

Although reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency. In this paper, we investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity.

Problem

• reinforcement learning
• large language models
• reasoning diversity
• semantic branching
• sample efficiency

Innovation

• BODHI-Trees
• semantic branching
• reinforcement learning with verifiable rewards
• reasoning diversity
• policy entropy