stable reinforcement learning

This work addresses the challenge in large language model (LLM) reinforcement learning where token-level surrogate objectives fail to effectively optimize sequence-level true rewards. We propose a first-order theoretical framework that, for the first time, formally characterizes the conditions under which token-level optimization is feasible and elucidates how training-inference mismatch and policy staleness impair training stability. Methodologically, we integrate importance sampling correction, gradient clipping, and routing-based replay to enable efficient online and offline policy updates within Mixture-of-Experts (MoE) architectures. Extensive experiments on a 30B-parameter MoE model demonstrate that our approach significantly improves training stability—remaining crash-free over hundreds of billions of GPU-hours—and achieves sustained, high-performance convergence.

Discrete-time reinforcement learning (RL) suffers from poor stability, slow convergence, and high sensitivity to time discretization when applied to continuous-time environments. To address these limitations, this paper proposes the Continuous-Time Deterministic Policy Gradient (CT-DDPG) algorithm. Methodologically, CT-DDPG is the first to incorporate the advantage function into a continuous-time deterministic policy gradient framework; it rigorously characterizes the advantage function via martingale theory under a continuous-time Markov process model, thereby establishing the first theoretical foundation for continuous-time policy gradients. This formulation bridges the stability gap between discrete- and continuous-time RL. Empirical evaluations across diverse control tasks demonstrate that CT-DDPG significantly improves convergence speed and robustness, while exhibiting strong insensitivity to time-step selection and environmental noise.

This work addresses the challenge of stabilizing complex, high-dimensional nonlinear systems under disturbances and environmental variations by introducing a novel paradigm termed “stable learning,” which prioritizes stability as the central objective through an integration of control theory and machine learning. Leveraging Lyapunov analysis, deep feature extraction, and data-driven feedback mechanisms, the authors formulate a unified six-tuple framework encompassing state space, metrics, and policy components, later extended to a seven-tuple model to accommodate constraints and tracking tasks. This framework systematically unifies eleven diverse problem classes across domains—including multi-agent cooperative tracking, visual servoing, board-game playing, and Push-T manipulation—distinguishing itself fundamentally from reinforcement learning and certificate-based approaches. Its broad applicability and efficacy are empirically validated across control, observation, and identification scenarios.

This work addresses the lack of a unified theoretical framework for reinforcement learning, which has hindered systematic analysis of its convergence, sample complexity, and generalization. Building upon Markov decision processes and Bellman operators, the paper introduces a cohesive analytical framework that integrates tools from operator theory, stochastic approximation, convex duality, and function approximation. This framework encompasses a broad range of algorithms, including value iteration, policy iteration, temporal difference methods, off-policy learning, and constrained MDPs. By leveraging contraction mappings, monotone operators, martingale techniques, mirror/proximal optimization, concentration inequalities, and mixing process theory, the study establishes finite-sample performance bounds and asymptotic convergence guarantees for diverse reinforcement learning algorithms, thereby forging a rigorous theoretical bridge between probability theory, optimization, and statistics.

Policy optimization algorithms suffer from poor interpretability and error-prone implementation due to the complexity of Markov decision process (MDP) modeling and inconsistent use of discounted versus average-reward settings. Method: This paper introduces a unified analytical framework that, for the first time, systematically integrates generalized ergodicity theory with perturbation analysis to characterize the steady-state behavior of diverse policy optimization algorithms under both discounted and average-reward criteria. Contribution/Results: The framework clarifies fundamental algorithmic principles, identifies and corrects common implementation pitfalls, and significantly enhances interpretability and robustness. Empirical validation on MDP modeling and linear quadratic regulator (LQR) benchmarks confirms the framework’s ability to capture algorithmic consistency. Quantitative analysis further demonstrates that minor adjustments to key design parameters exert decisive influence on convergence properties and performance.

This work investigates the instability of reinforcement learning (RL) compared to supervised fine-tuning (SFT) in large language model training. From a gradient perspective, it identifies—for the first time—that the convexity of the logits space plays a pivotal role in the stability of policy optimization. Building on this insight, the authors propose the Logits Convex Optimization (LCO) framework, which aligns and optimizes objectives directly in the logits space by integrating convexity-aware alignment, gradient direction analysis, and enhanced policy gradient estimation. This approach substantially improves training stability without sacrificing performance. Extensive experiments across multiple model families and benchmarks demonstrate that LCO consistently outperforms conventional RL methods, achieving superior stability and competitive or better task performance.

In safe reinforcement learning (SRL), the selection and update of the Lagrange multiplier λ lack theoretical grounding and empirical validation; λ is highly sensitive, and automatic updates often suffer from oscillation, undermining algorithmic stability and performance. To address this, we propose the λ-profile visualization technique to demonstrate that the optimal λ* admits no universal intuitive rule. We further design an adaptive multiplier update framework integrating Lagrangian dual optimization with PID control. Our method is rigorously evaluated across multiple SRL benchmarks. Experiments show that automated λ adaptation not only surpasses performance achieved with a fixed optimal λ but also yields smoother, more stable learning trajectories. While PID control effectively suppresses oscillations, it entails a trade-off between robustness and hyperparameter tuning overhead. The implementation is open-sourced, establishing a new empirical and methodological paradigm for studying constraint optimization stability in SRL.

This work addresses the convergence instability of linear Q-learning under general conditions, where existing theory lacks rigorous analysis of the stabilizing mechanisms in periodic hard and soft target updates. By modeling the dynamics induced by the Bellman optimality operator as a switched linear system, the paper establishes, for the first time, a convergence criterion for target-update schemes based on the joint spectral radius. It rigorously proves that both hard and soft target updates converge to the exact projected Q-Bellman solution under explicit spectral conditions and stepsize constraints. The framework is further extended to reinforcement learning settings with stochastic noise, providing a unified theoretical foundation for the stability of Q-learning with linear function approximation.

This work addresses the challenge posed by time-varying system dynamics—such as those caused by wear or operating condition drift—that render conventional reinforcement learning methods ineffective. To tackle this issue, the paper proposes a model-based reinforcement learning approach tailored for non-stationary environments. The method integrates a Gaussian process dynamics model with an adaptive data buffering mechanism that explicitly limits the influence of outdated experiences, thereby enabling well-calibrated uncertainty estimation. Notably, it is the first to incorporate variable-budget non-stationarity analysis into a model-based control framework, offering dynamic regret guarantees. Empirical evaluations on multiple time-varying continuous control benchmarks demonstrate that the proposed algorithm significantly outperforms existing approaches, confirming its robustness and efficiency in non-stationary settings.

This work addresses the lack of theoretical guarantees for Feudal Q-learning in hierarchical reinforcement learning. Methodologically, it establishes the first rigorous framework for convergence and stability analysis by integrating stochastic approximation theory, ordinary differential equation (ODE) dynamical modeling, and a game-theoretic perspective—formulating hierarchical policy updates as an equilibrium evolution process in a multi-agent game. Theoretically, under standard Markov assumptions and diminishing step-size conditions, the algorithm is proven to converge almost surely to a stable equilibrium, which corresponds to a Nash equilibrium between subgoal policies and high-level guidance policies. Empirical evaluations validate the predicted dynamical behavior and convergence rates, significantly enhancing the interpretability and reliability of hierarchical RL. This work provides the first formal theoretical foundation for Feudal Q-learning and advances the deep integration of game theory and hierarchical reinforcement learning.