Statistical Learning of Distributionally Robust Stochastic Control in Continuous State Spaces

2024-06-17

International Conference on Artificial Intelligence and Statistics

AI Summary

This paper addresses distributionally robust stochastic control in continuous state spaces to mitigate the policy fragility of conventional i.i.d. Markov models—arising from neglecting environmental input distribution shifts and endogenous dependencies. We propose a novel paradigm that balances modeling simplicity and robustness: adaptive adversarial perturbations are embedded within dynamic programming to unify *f*-divergence and Wasserstein-type ambiguity sets. We establish, for the first time in continuous spaces, a unified learning theory for robust value functions under both ambiguity-set classes, providing finite-sample minimax convergence rate bounds. Integrating distributionally robust optimization, stochastic control, and nonparametric statistics, we design a computationally tractable minimax policy learning algorithm. Experiments demonstrate that the framework achieves both statistical efficiency and strong robustness across real-world applications—including supply chain management and finance.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Reasoning under Uncertainty: Stochastic Optimization
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We explore the control of stochastic systems with potentially continuous state and action spaces, characterized by the state dynamics $X_{t+1} = f(X_t, A_t, W_t)$. Here, $X$, $A$, and $W$ represent the state, action, and exogenous random noise processes, respectively, with $f$ denoting a known function that describes state transitions. Traditionally, the noise process ${W_t, t geq 0}$ is assumed to be independent and identically distributed, with a distribution that is either fully known or can be consistently estimated. However, the occurrence of distributional shifts, typical in engineering settings, necessitates the consideration of the robustness of the policy. This paper introduces a distributionally robust stochastic control paradigm that accommodates possibly adaptive adversarial perturbation to the noise distribution within a prescribed ambiguity set. We examine two adversary models: current-action-aware and current-action-unaware, leading to different dynamic programming equations. Furthermore, we characterize the optimal finite sample minimax rates for achieving uniform learning of the robust value function across continuum states under both adversary types, considering ambiguity sets defined by $f_k$-divergence and Wasserstein distance. Finally, we demonstrate the applicability of our framework across various real-world settings.

Problem

• Learning robust stochastic control for continuous state-action spaces with data-driven approaches
• Addressing policy fragility through distributionally robust adversarial perturbation models
• Developing minimax statistical rates and deep RL algorithms for robust policy optimization

Innovation

• Distributionally robust stochastic control with adversarial perturbations
• Optimal finite-sample minimax rates for robust learning
• Deep reinforcement learning algorithms for policy computation