Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamics

2026-02-06

AI Summary

This work addresses the significant performance degradation of reinforcement learning policies when deployed in dynamic, non-stationary real-world environments. To tackle this challenge, the authors propose a lightweight online adaptation framework that uniquely integrates Echo State Networks (ESNs) with Recursive Least Squares (RLS). The ESN encodes recent observations into a contextual representation, while RLS enables rapid online updates of the readout weights—eliminating the need for backpropagation, pretraining, or privileged information. This approach achieves stable adaptation within just a few control steps, even under severe out-of-distribution shifts or intra-episode environmental changes. Empirical results on CartPole and HalfCheetah demonstrate substantial improvements over domain randomization and state-of-the-art adaptive baselines, highlighting its suitability for edge devices and real-world robotic control.

Technology Category

• Machine Learning: Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement learning (RL) policies trained in simulation often suffer from severe performance degradation when deployed in real-world environments due to non-stationary dynamics. While Domain Randomization (DR) and meta-RL have been proposed to address this issue, they typically rely on extensive pretraining, privileged information, or high computational cost, limiting their applicability to real-time and edge systems. In this paper, we propose a lightweight online adaptation framework for RL based on Reservoir Computing. Specifically, we integrate an Echo State Networks (ESNs) as an adaptation module that encodes recent observation histories into a latent context representation, and update its readout weights online using Recursive Least Squares (RLS). This design enables rapid adaptation without backpropagation, pretraining, or access to privileged information. We evaluate the proposed method on CartPole and HalfCheetah tasks with severe and abrupt environment changes, including periodic external disturbances and extreme friction variations. Experimental results demonstrate that the proposed approach significantly outperforms DR and representative adaptive baselines under out-of-distribution dynamics, achieving stable adaptation within a few control steps. Notably, the method successfully handles intra-episode environment changes without resetting the policy. Due to its computational efficiency and stability, the proposed framework provides a practical solution for online adaptation in non-stationary environments and is well suited for real-world robotic control and edge deployment.

Problem

• non-stationary dynamics
• reinforcement learning
• online adaptation
• real-world deployment
• environmental changes

Innovation

• Echo State Networks
• Online Adaptation
• Reinforcement Learning
• Non-Stationary Dynamics
• Recursive Least Squares