🤖 AI Summary
Traditional robust control theory fails in reinforcement learning settings due to gradient uncertainty introduced when approximating value function gradients, undermining stability and performance guarantees.
Method: We propose the first robust control framework that explicitly models gradient uncertainty. By formulating a zero-sum dynamic game integrating system dynamics and gradient perturbations, we expose the mismatch mechanism of quadratic value function assumptions under nonzero gradient uncertainty. This yields a generalized Hamilton–Jacobi–Bellman–Isaacs (GU-HJBI) equation featuring non-polynomial correction terms and an associated nonlinear optimal control law. Using viscosity solution theory and uniform ellipticity, we establish a comparison principle; further, we integrate perturbation analysis with an Actor–Critic architecture to design the Gradient-Uncertainty-Robust Actor–Critic (GURAC) algorithm.
Results: We theoretically guarantee well-posedness of the GU-HJBI equation, and empirical evaluations demonstrate significantly improved training stability and robustness.
📝 Abstract
We introduce a novel extension to robust control theory that explicitly addresses uncertainty in the value function's gradient, a form of uncertainty endemic to applications like reinforcement learning where value functions are approximated. We formulate a zero-sum dynamic game where an adversary perturbs both system dynamics and the value function gradient, leading to a new, highly nonlinear partial differential equation: the Hamilton-Jacobi-Bellman-Isaacs Equation with Gradient Uncertainty (GU-HJBI). We establish its well-posedness by proving a comparison principle for its viscosity solutions under a uniform ellipticity condition. Our analysis of the linear-quadratic (LQ) case yields a key insight: we prove that the classical quadratic value function assumption fails for any non-zero gradient uncertainty, fundamentally altering the problem structure. A formal perturbation analysis characterizes the non-polynomial correction to the value function and the resulting nonlinearity of the optimal control law, which we validate with numerical studies. Finally, we bridge theory to practice by proposing a novel Gradient-Uncertainty-Robust Actor-Critic (GURAC) algorithm, accompanied by an empirical study demonstrating its effectiveness in stabilizing training. This work provides a new direction for robust control, holding significant implications for fields where function approximation is common, including reinforcement learning and computational finance.