online learning

This work addresses the instability and uncontrolled update magnitudes in streaming reinforcement learning caused by fixed step sizes. Inspired by the normalized least mean squares (NLMS) algorithm, we propose a novel approach that adaptively determines step sizes by inversely reasoning from desired update effects. Extending this principle to deep reinforcement learning, we introduce Intentional TD and Intentional Policy Gradient algorithms, which incorporate eligibility traces and diagonal scaling to enable stable and efficient learning in a purely streaming setting—without relying on experience replay buffers. Empirical results demonstrate that our method achieves state-of-the-art streaming performance across multiple benchmark tasks, matching or surpassing batch-based algorithms that depend on experience replay.

This work addresses the long-standing challenge of credit assignment over extended sequences in streaming reinforcement learning under partial observability, where truncated backpropagation through time (BPTT) severely limits gradient horizons. The authors propose a novel streaming reinforcement learning method based on Recurrent Trace Units (RTUs), which for the first time enables exact real-time recurrent learning (RTRL) without replay buffers or batch updates. Leveraging a diagonal recurrent architecture, the approach incurs only linear time and memory complexity. It is compatible with both discrete and continuous control and supports online policy optimization. Empirical results demonstrate that the method significantly outperforms existing streaming baselines on MemoryChain, POPGym, and partially observable MuJoCo tasks, while closely approaching the performance of batch-based PPO.

This work addresses the challenge of continual learning in dynamic environments characterized by non-stationary data streams by proposing Streamed Continual Learning (SCL), a novel framework that unifies the paradigms of continual learning and streaming machine learning for the first time. SCL integrates knowledge retention mechanisms from continual learning with online updating strategies from streaming learning to construct an efficient and adaptive unified architecture. The study delineates the core characteristics and technical pathways of SCL, thereby fostering synergistic innovation between the two fields and laying a theoretical foundation for developing general-purpose adaptive intelligent systems capable of operating effectively in dynamic environments.

This paper addresses real-time model tracking under time-varying optimization in online learning from streaming data. Unlike conventional static optimization, we propose a structured weight-based modeling framework that explicitly formulates the objective as a weighted average of historical losses. We systematically analyze two weighting schemes: uniform and geometrically discounted. Theoretically, we derive the first tight tracking error bounds: under uniform weighting, the error converges to zero at rate $O(1/t)$; under geometric discounting, it converges to a controllable nonzero steady-state error. Algorithmically, we integrate gradient-based updates to enable efficient online adaptation. Numerical experiments validate both the theoretical bounds and the empirical effectiveness of the proposed approach.

Existing deep policy gradient methods rely on large replay buffers, batch updates, or target networks, hindering efficient online incremental learning on resource-constrained real-world robots. To address this, we propose the first purely incremental Action Value Gradient (AVG) algorithm—eliminating both replay buffers and target networks—and performing parameter updates using only single-step transitions. We further introduce online normalization and dynamic scaling compensation to mitigate gradient instability and training collapse under minimal buffering. In simulation benchmarks, AVG matches the performance of leading batch-based methods. Crucially, we present the first empirical validation of stable, efficient deep reinforcement learning with single-step incremental updates on real robotic platforms—including a 7-DOF manipulator and a mobile robot—demonstrating robust online adaptive control under stringent computational and memory constraints. This work establishes a new paradigm for resource-efficient, truly online deep RL in embedded robotic systems.

This work addresses the instability inherent in streaming reinforcement learning settings characterized by the absence of replay buffers and purely online training. Through a systematic analysis of algorithms such as DQN and C51 under this paradigm, the study identifies gradient boundedness and variance adaptivity as critical factors for stable training. Building on these insights, the authors propose Adaptive Q(λ), a novel algorithm that integrates eligibility traces, a distributional reinforcement learning framework, and a variance-adaptive update mechanism, optimized via the Adam optimizer. Empirical evaluation demonstrates that C51 achieves performance comparable to StreamQ across 55 Atari games, while Adaptive Q(λ) attains an average score nearly twice the human baseline, substantially outperforming existing approaches.

This work addresses catastrophic forgetting in continual learning under non-stationary data streams by proposing the COLD framework, which introduces, for the first time, the Drift-Plus-Penalty stochastic optimization method from control theory into this domain. COLD formulates forgetting as a controlled dynamic process, employing virtual queues to track performance deviations on historical tasks and jointly minimizing the current task loss and queue drift at each optimization step. This mechanism explicitly governs the stability-plasticity trade-off. The framework provides theoretical guarantees on stability and convergence, and achieves significantly superior performance over state-of-the-art methods on standard benchmarks, enabling controllable and efficient suppression of catastrophic forgetting.

This work addresses the challenge in offline-to-online reinforcement learning where static data mixing strategies often induce distributional shift, compromising both training stability and asymptotic performance. The authors propose ROAD, a novel framework that formulates data mixing as a bilevel optimization problem: the outer loop optimizes policy performance by dynamically adjusting the mixing strategy via a multi-armed bandit mechanism, while the inner loop performs Q-learning updates augmented with surrogate gradient estimation, offline prior preservation, and overestimation suppression techniques. ROAD introduces a plug-and-play adaptive replay mechanism that automatically adapts to diverse environments without manual hyperparameter tuning. Empirical results demonstrate that ROAD significantly outperforms existing replay strategies across multiple datasets, achieving superior final performance while maintaining training stability.

This work addresses the limitations of existing streaming video understanding methods, which predominantly follow a “watch-then-answer” paradigm and struggle to evaluate a model’s ability to make timely and accurate active decisions under partial observation, compounded by sparse supervision and joint optimization challenges. To overcome these issues, the authors propose StreamPro, a two-stage training framework that introduces CB-Stream Loss to mitigate response signal imbalance and a novel Group Relative Policy Optimization (GRPO) algorithm that integrates round-level and trajectory-level rewards, enabling a shift from passive perception to active decision-making. Additionally, they construct StreamPro-Bench—the first benchmark specifically designed for streaming video understanding—spanning three dimensions: perceptual comprehension, temporal reasoning, and active decision-making. Experiments show that StreamPro achieves a score of 41.5 on StreamPro-Bench, substantially outperforming the previous best result of 10.4, and attains a strong performance of 78.9 on StreamingBench-RTVU.

This work addresses the data efficiency bottleneck of reinforcement learning in high-cost, slow-interaction settings such as commercial and medical applications. It introduces large deviation theory into data collection design, proposing the exponential decay rate of policy selection error probability as a novel efficiency metric. Building on this foundation, the paper develops a unified optimization framework to characterize optimal data acquisition strategies and introduces two complementary notions of optimality. To enhance scalability, the authors devise an adaptive sampling algorithm that combines convex relaxation with a lazy one-step projected subgradient method, offering provably near-optimal performance. Theoretical analysis establishes its approximate robust optimality, while numerical experiments demonstrate substantially improved data efficiency.