FOSP: Fine-tuning Offline Safe Policy through World Models

2024-07-06

arXiv.org

AI Summary

To address the weak offline policy generalization and inefficient online safe exploration of vision-based robots under unknown safety constraints, this paper proposes a novel “offline pretraining + online safe fine-tuning” paradigm. Methodologically, it introduces (1) a world-model-driven offline policy pretraining framework to enhance data efficiency and initial safety, and (2) a reachability-guided in-model optimization and safe policy expansion mechanism enabling in-sample safe fine-tuning. This work is the first to systematically bridge the safety-aware generalization gap from offline to online reinforcement learning. Evaluated on five vision-only simulated tasks and real-robot deployment with limited data, the approach significantly improves safety constraint satisfaction rates and cross-scenario generalization performance, effectively mitigating offline policies’ strong dependence on static datasets.

Technology Category

• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Offline Safe Reinforcement Learning (RL) seeks to address safety constraints by learning from static datasets and restricting exploration. However, these approaches heavily rely on the dataset and struggle to generalize to unseen scenarios safely. In this paper, we aim to improve safety during the deployment of vision-based robotic tasks through online fine-tuning an offline pretrained policy. To facilitate effective fine-tuning, we introduce model-based RL, which is known for its data efficiency. Specifically, our method employs in-sample optimization to improve offline training efficiency while incorporating reachability guidance to ensure safety. After obtaining an offline safe policy, a safe policy expansion approach is leveraged for online fine-tuning. The performance of our method is validated on simulation benchmarks with five vision-only tasks and through real-world robot deployment using limited data. It demonstrates that our approach significantly improves the generalization of offline policies to unseen safety-constrained scenarios. To the best of our knowledge, this is the first work to explore offline-to-online RL for safe generalization tasks.

Problem

• Improves safety in vision-based robotic tasks deployment.
• Enhances generalization of offline policies to unseen scenarios.
• Introduces model-based RL for efficient offline-to-online fine-tuning.

Innovation

• Model-based RL for data efficiency
• In-sample optimization for offline training
• Safe policy expansion for online fine-tuning