Deep SPI: Safe Policy Improvement via World Models

2025-10-14

AI Summary

This paper addresses safe policy improvement (SPI) in online deep reinforcement learning, proposing DeepSPI—the first algorithm to extend classical offline SPI theory to general online settings. Methodologically, DeepSPI integrates world modeling and representation learning, incorporating local transition/reward loss constraints and a regularization mechanism for policy updates, thereby ensuring monotonic policy improvement and convergence under model-free online interaction. Theoretically, it establishes the first provably safe online deep SPI framework with formal safety guarantees. Empirically, DeepSPI achieves performance on par with or surpassing strong baselines—including PPO and DeepMDPs—on the ALE-57 benchmark, while maintaining rigorous theoretical safety properties.

Technology Category

Machine Learning: Online Learning & BanditsSearch and Optimization: Learning to SearchNatural Language Processing: Safety and Robustness

Application Category

Responsible Web: Human-perceived consequences of algorithmic deployment on the webSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Safe policy improvement (SPI) offers theoretical control over policy updates, yet existing guarantees largely concern offline, tabular reinforcement learning (RL). We study SPI in general online settings, when combined with world model and representation learning. We develop a theoretical framework showing that restricting policy updates to a well-defined neighborhood of the current policy ensures monotonic improvement and convergence. This analysis links transition and reward prediction losses to representation quality, yielding online, "deep" analogues of classical SPI theorems from the offline RL literature. Building on these results, we introduce DeepSPI, a principled on-policy algorithm that couples local transition and reward losses with regularised policy updates. On the ALE-57 benchmark, DeepSPI matches or exceeds strong baselines, including PPO and DeepMDPs, while retaining theoretical guarantees.

Problem

• Extends safe policy improvement guarantees to online deep reinforcement learning
• Links representation quality to transition and reward prediction losses
• Develops theoretically-grounded algorithm maintaining guarantees while matching baselines

Innovation

• World models ensure safe online policy improvement
• Regularized updates restrict policy neighborhood for guarantees
• Transition and reward losses link to representation quality