Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors

2026-09-28

AI Summary

This study addresses the challenge that high-dimensional parameter updates in Reinforcement Learning with Verifiable Rewards (RLVR) for large language models complicate dynamic analysis and frequently induce early training collapse. To tackle this, we present the first quantitative characterization of the RLVR geometric structure, leveraging vector steering to reveal a low-dimensional effective manifold within the activation space along with its capacity constraints and principal subspace separation properties. Building upon these insights, we propose Alpha-Stabler, a plug-and-play framework that enables early collapse warning through principal component monitoring and enforces stabilization via gradient orthogonal projection. Extensive evaluations across five large language models and six tasks demonstrate that Alpha-Stabler significantly stabilizes 2000-step training processes while consistently enhancing reinforcement learning gains.

Abstract

Reinforcement learning (RL) has become a key paradigm for enhancing the reasoning of large language models, yet the high dimensionality of parameter updates makes its training dynamics hard to analyze. We study reinforcement learning with verifiable rewards (RLVR) and use vector steering to identify a low-dimensional effective manifold in activation space associated with RL-induced gains. We uncover two geometric properties. (1) Effective Manifold Capacity: the capacity needed to reproduce RL gains can be very small but is not infinitely compressible; at extremely low capacity, intervention dimensionality and input-dependent expressiveness become key constraints, and this requirement varies with injection depth. (2) Control Manifold Separation: effective control directions lie mainly in the low-variance complement of the activation principal subspace. Within a task and base model, the learned geometry stays largely consistent across training configurations, and across tasks geometric alignment correlates with capability transfer. Experiments on 5 LLMs and 6 verifiable-reward tasks support these findings. We then propose Alpha-Stabler, a plug-and-play framework with a Predictor that monitors principal-subspace intrusion for early collapse warnings, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving the orthogonal complement. Alpha-Stabler stabilizes training for 2,000 steps and consistently improves RL gains, offering practical insights for robust post-training. Code: https://github.com/caiyuchen-ustc/On_Policy_Vector_Training

Problem

• Reinforcement Learning
• Large Language Models
• RLVR
• Training Dynamics
• Geometry

Innovation

• Reinforcement Learning with Verifiable Rewards
• Vector Steering
• Low-dimensional Effective Manifold
• Control Manifold Separation
• Alpha-Stabler