🤖 AI Summary
In infinite-horizon reinforcement learning, statistical inference for policy value functions remains challenging—particularly under high-dimensional state spaces, non-unique optimal policies, and infinitely many decision points—making reliable confidence interval construction difficult. To address this, we propose the first series-expansion-based Q-function modeling framework and the recursive Series-based Adaptive Value Estimation (SAVE) method. SAVE integrates sieve estimation with asymptotic statistical inference to ensure nominal coverage probability under policy-dependent data. We establish theoretical guarantees showing that the constructed confidence intervals achieve asymptotically exact coverage. Simulation studies demonstrate robustness to model misspecification and policy variation. Empirical evaluation on a mobile health dataset confirms the statistically significant improvement in patient health outcomes attributable to the RL intervention.
📝 Abstract
Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision‐making problems. The goodness of a policy is measured by its value function starting from some initial state. The focus of this paper was to construct confidence intervals (CIs) for a policy’s value in infinite horizon settings where the number of decision points diverges to infinity. We propose to model the action‐value state function (Q‐function) associated with a policy based on series/sieve method to derive its confidence interval. When the target policy depends on the observed data as well, we propose a SequentiAl Value Evaluation (SAVE) method to recursively update the estimated policy and its value estimator. As long as either the number of trajectories or the number of decision points diverges to infinity, we show that the proposed CI achieves nominal coverage even in cases where the optimal policy is not unique. Simulation studies are conducted to back up our theoretical findings. We apply the proposed method to a dataset from mobile health studies and find that reinforcement learning algorithms could help improve patient’s health status. A Python implementation of the proposed procedure is available at https://github.com/shengzhang37/SAVE.