Sample-Efficient Online Control Policy Learning with Real-Time Recursive Model Updates

2025-09-09

AI Summary

To address low sample efficiency, model update latency, and high computational overhead in data-driven control under dynamic environments, this paper proposes Recursive Koopman Learning (RKL). RKL leverages Koopman operator theory to construct linear observable models of nonlinear systems and introduces an optimization-friendly recursive update mechanism, achieving O(1) time complexity independent of dataset size—enabling real-time online learning and lightweight deployment. Implemented efficiently in C++, RKL is validated in both simulation and on a soft robotic hardware platform. Results demonstrate that RKL surpasses state-of-the-art baselines using less than 10% of their training data, while significantly improving control accuracy, stability, and convergence speed. The framework thus provides a scalable, low-latency solution for adaptive control in resource-constrained settings.

Technology Category

• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Learning to Search
• Machine Learning: Life-Long and Continual Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Data-driven control methods need to be sample-efficient and lightweight, especially when data acquisition and computational resources are limited -- such as during learning on hardware. Most modern data-driven methods require large datasets and struggle with real-time updates of models, limiting their performance in dynamic environments. Koopman theory formally represents nonlinear systems as linear models over observables, and Koopman representations can be determined from data in an optimization-friendly setting with potentially rapid model updates. In this paper, we present a highly sample-efficient, Koopman-based learning pipeline: Recursive Koopman Learning (RKL). We identify sufficient conditions for model convergence and provide formal algorithmic analysis supporting our claim that RKL is lightweight and fast, with complexity independent of dataset size. We validate our method on a simulated planar two-link arm and a hybrid nonlinear hardware system with soft actuators, showing that real-time recursive Koopman model updates improve the sample efficiency and stability of data-driven controller synthesis -- requiring only <10% of the data compared to benchmarks. The high-performance C++ codebase is open-sourced. Website: https://www.zixinatom990.com/home/robotics/corl-2025-recursive-koopman-learning.

Problem

• Developing sample-efficient online control policy learning
• Enabling real-time recursive model updates for dynamic environments
• Reducing data requirements for data-driven controller synthesis

Innovation

• Recursive Koopman Learning for control
• Real-time recursive model updates
• Lightweight complexity independent of data size