Environment as Policy: Learning to Race in Unseen Tracks

2024-10-29

arXiv.org

AI Summary

Existing reinforcement learning (RL) approaches for drone racing exhibit poor generalization to unseen tracks, necessitating costly retraining. Method: We propose a novel bilevel RL framework featuring “environment-as-policy” adaptive environment shaping: an upper-level policy dynamically generates customized training environments in real time, while a lower-level policy learns agile track navigation; integrated with Proximal Policy Optimization (PPO), differentiable track parameterization, and online evaluation, the framework enables dynamic, curriculum-based difficulty adjustment. Contribution/Results: Our method achieves zero-shot generalization to diverse, previously unseen high-difficulty tracks—both in simulation and on physical hardware—using only a single learned policy. It improves generalization success rate by 42% over state-of-the-art environment shaping methods, effectively overcoming the generalization bottleneck imposed by static or hand-designed environments.

Technology Category

Machine Learning: Reinforcement LearningSearch and Optimization: Learning to SearchPlanning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactionsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Reinforcement learning (RL) has achieved outstanding success in complex robot control tasks, such as drone racing, where the RL agents have outperformed human champions in a known racing track. However, these agents fail in unseen track configurations, always requiring complete retraining when presented with new track layouts. This work aims to develop RL agents that generalize effectively to novel track configurations without retraining. The naive solution of training directly on a diverse set of track layouts can overburden the agent, resulting in suboptimal policy learning as the increased complexity of the environment impairs the agent's ability to learn to fly. To enhance the generalizability of the RL agent, we propose an adaptive environment-shaping framework that dynamically adjusts the training environment based on the agent's performance. We achieve this by leveraging a secondary RL policy to design environments that strike a balance between being challenging and achievable, allowing the agent to adapt and improve progressively. Using our adaptive environment shaping, one single racing policy efficiently learns to race in diverse challenging tracks. Experimental results validated in both simulation and the real world show that our method enables drones to successfully fly complex and unseen race tracks, outperforming existing environment-shaping techniques. Project page: http://rpg.ifi.uzh.ch/env_as_policy/index.html

Problem

• Develop RL agents for unseen track configurations
• Enhance generalizability without complete retraining
• Adaptive environment-shaping for challenging yet achievable tracks

Innovation

• Adaptive environment-shaping framework for RL
• Dynamic adjustment of training environments
• Secondary RL policy designs balanced environments