Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models

2026-07-28

AI Summary

This study investigates why reinforcement learning (RL) fine-tuning outperforms supervised fine-tuning (SFT) in mathematical reasoning, despite the underlying mechanisms remaining unclear. Through systematic analyses—including linear probing, inter-layer mean ablation, and token-count variability under repeated sampling—the authors compare the internal representational structures and computational resource allocation strategies of RL- and SFT-trained models. The findings reveal that RL models develop more linearly separable and hierarchically organized representations, with deeper layers contributing more significantly to performance. Moreover, the work uncovers a novel link between token allocation variability and policy stability, demonstrating that certain RL models adaptively modulate computational resources to sustain consistent reasoning. These insights offer a new perspective on how RL enhances reasoning capabilities in language models.

Abstract

Large reasoning models trained via reinforcement learning (RL) have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear. We therefore ask, what internal representational differences enable RL models' superior performance? Our work presents two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations. Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers. Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems. Finally, we analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation. While we observe higher variability in some RL-tuned models than in their SFT counterparts, we see strong consistency in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone. We believe this token-allocation variability reveals the spread of plausible on-policy reasoning, highlighting which models exhibit stable policies versus those that are under-determined, potentially non-identifiable solution behaviour.

Problem

• reasoning performance
• representational quality
• reinforcement learning
• supervised fine-tuning
• mathematical problem-solving

Innovation

• reinforcement learning
• representational structure
• linear probing
• mean ablation
• adaptive compute allocation