Kernel-Based Optimal Control: An Infinitesimal Generator Approach

2024-12-02

arXiv.org

AI Summary

This work addresses data-driven optimal control for nonlinear stochastic systems. We propose an operator-theoretic framework grounded in reproducing kernel Hilbert spaces (RKHS), which requires only finite samples of system dynamics and stage costs, along with control constraints. The method directly learns the infinitesimal generator of the controlled stochastic diffusion process and—uniquely—embeds it into an RKHS, enabling seamless integration with convex operator-valued Hamilton–Jacobi–Bellman (HJB) recursion. Our key contribution is a nonparametric kernel estimator for the uncontrolled generator, unifying data-driven modeling with rigorous theoretical optimality guarantees. Evaluated on synthetic stochastic differential equations and simulated robotic control tasks, the approach substantially outperforms state-of-the-art data-driven and classical nonlinear programming methods, achieving high accuracy, strong generalization, and strict theoretical consistency.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper presents a novel operator-theoretic approach for optimal control of nonlinear stochastic systems within reproducing kernel Hilbert spaces. Our learning framework leverages data samples of system dynamics and stage cost functions, with only control penalties and constraints provided. The proposed method directly learns the infinitesimal generator of a controlled stochastic diffusion in an infinite-dimensional hypothesis space. We demonstrate that our approach seamlessly integrates with modern convex operator-theoretic Hamilton-Jacobi-Bellman recursions, enabling a data-driven solution to the optimal control problems. Furthermore, our learning framework includes nonparametric estimators for uncontrolled infinitesimal generators as a special case. Numerical experiments, ranging from synthetic differential equations to simulated robotic systems, showcase the advantages of our approach compared to both modern data-driven and classical nonlinear programming methods for optimal control.

Problem

• Optimal control of nonlinear stochastic systems using kernel methods
• Data-driven learning of infinitesimal generators for controlled diffusions
• Integration with convex operator-theoretic HJB recursions for control solutions

Innovation

• Operator-theoretic approach in kernel spaces
• Learns infinitesimal generator from data
• Integrates with convex HJB recursions