Wavelet Predictive Representations for Non-Stationary Reinforcement Learning

2025-10-06

AI Summary

Real-world environments exhibit strong non-stationarity, making it difficult for agents to rapidly adapt to dynamically evolving MDP sequences. Existing non-stationary reinforcement learning (NSRL) methods often assume regular, predictable task evolution and thus suffer from limited generalization under highly stochastic or abrupt environmental shifts. To address this, we propose WISDOM—a novel NSRL framework that pioneers the integration of wavelet analysis into reinforcement learning. WISDOM employs wavelet transforms to extract multi-scale trends and transient change features from task sequences, yielding predictive task representations. It introduces a wavelet-domain temporal difference update operator for fine-grained modeling of MDP evolution, with theoretical convergence guarantees. Furthermore, it unifies autoregressive modeling with multi-scale representation learning. Evaluated on multiple non-stationary benchmarks, WISDOM achieves significant improvements in sample efficiency and asymptotic performance, demonstrating robust adaptation capability in complex dynamic environments.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Multiagent Learning
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The real world is inherently non-stationary, with ever-changing factors, such as weather conditions and traffic flows, making it challenging for agents to adapt to varying environmental dynamics. Non-Stationary Reinforcement Learning (NSRL) addresses this challenge by training agents to adapt rapidly to sequences of distinct Markov Decision Processes (MDPs). However, existing NSRL approaches often focus on tasks with regularly evolving patterns, leading to limited adaptability in highly dynamic settings. Inspired by the success of Wavelet analysis in time series modeling, specifically its ability to capture signal trends at multiple scales, we propose WISDOM to leverage wavelet-domain predictive task representations to enhance NSRL. WISDOM captures these multi-scale features in evolving MDP sequences by transforming task representation sequences into the wavelet domain, where wavelet coefficients represent both global trends and fine-grained variations of non-stationary changes. In addition to the auto-regressive modeling commonly employed in time series forecasting, we devise a wavelet temporal difference (TD) update operator to enhance tracking and prediction of MDP evolution. We theoretically prove the convergence of this operator and demonstrate policy improvement with wavelet task representations. Experiments on diverse benchmarks show that WISDOM significantly outperforms existing baselines in both sample efficiency and asymptotic performance, demonstrating its remarkable adaptability in complex environments characterized by non-stationary and stochastically evolving tasks.

Problem

• Addresses non-stationary reinforcement learning in dynamic environments
• Captures multi-scale environmental changes using wavelet analysis
• Enhances adaptability to stochastically evolving Markov Decision Processes

Innovation

• Wavelet-domain predictive task representations for NSRL
• Wavelet temporal difference update operator for MDP evolution
• Multi-scale feature capture in non-stationary MDP sequences