Testing Stationarity and Change Point Detection in Reinforcement Learning

2022-03-03

AI Summary

Offline reinforcement learning (RL) in nonstationary environments remains challenging due to the absence of online interaction and strong assumptions of environmental stationarity. Method: This paper proposes a Q-function stationarity test and sequential change-point detection framework operating solely on offline data. It introduces the first consistent nonstationarity test for Q-functions, integrating CUSUM-type statistics, adaptive sliding windows, Bellman error modeling, and nonparametric kernel estimation, with rigorous asymptotic theoretical guarantees. The framework is plug-and-play compatible with mainstream RL algorithms. Contribution/Results: We establish consistency of the stationarity test and localization consistency of detected change points. Empirical evaluation across traffic signal control, robotic simulation, and real-world 2018 Intern Health data demonstrates substantial improvements: policy robustness increases significantly, and latency in responding to environmental shifts decreases by over 37%. The method eliminates reliance on environment stationarity while requiring no online exploration.

Abstract

We consider offline reinforcement learning (RL) methods in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the system transition and the reward function to be constant over time. However, the stationarity assumption is restrictive in practice and is likely to be violated in a number of applications, including traffic signal control, robotics and mobile health. In this paper, we develop a consistent procedure to test the nonstationarity of the optimal Q-function based on pre-collected historical data, without additional online data collection. Based on the proposed test, we further develop a sequential change point detection method that can be naturally coupled with existing state-of-the-art RL methods for policy optimization in nonstationary environments. The usefulness of our method is illustrated by theoretical results, simulation studies, and a real data example from the 2018 Intern Health Study. A Python implementation of the proposed procedure is available at https://github.com/limengbinggz/CUSUM-RL.

Problem

• Adaptive Learning
• Environmental Changes
• Optimization Strategies

Innovation

• Offline Learning
• Adaptive Strategy Adjustment
• Dynamic Environment