Sim-to-Real Transfer in Reinforcement Learning for Maneuver Control of a Variable-Pitch MAV

2025-04-10

AI Summary

Variable-pitch micro air vehicles (VPP MAVs) face significant challenges in sim-to-real transfer for reinforcement learning and struggle to achieve high-agility aerial acrobatic control. Method: This paper proposes a real-simulation co-adaptive transfer framework integrating system identification, domain randomization, and curriculum learning, coupled with a cascaded control architecture that decouples high-level policy learning from low-level fast dynamics response. Contribution/Results: The framework achieves, for the first time, zero-shot physical deployment of VPP MAVs without any fine-tuning on real-world data. Experiments demonstrate successful execution of complex maneuvers—including barrel rolls and wall-rebound turns—validating substantial improvements in sim-to-real transfer efficiency and control robustness. This work provides a scalable methodological foundation for intelligent control of highly dynamic rotorcraft platforms.

Technology Category

• Intelligent Robots: Behavior Learning & Control
• Multiagent Systems: Adversarial Agents
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reinforcement learning (RL) algorithms can enable high-maneuverability in unmanned aerial vehicles (MAVs), but transferring them from simulation to real-world use is challenging. Variable-pitch propeller (VPP) MAVs offer greater agility, yet their complex dynamics complicate the sim-to-real transfer. This paper introduces a novel RL framework to overcome these challenges, enabling VPP MAVs to perform advanced aerial maneuvers in real-world settings. Our approach includes real-to-sim transfer techniques-such as system identification, domain randomization, and curriculum learning to create robust training simulations and a sim-to-real transfer strategy combining a cascade control system with a fast-response low-level controller for reliable deployment. Results demonstrate the effectiveness of this framework in achieving zero-shot deployment, enabling MAVs to perform complex maneuvers such as flips and wall-backtracking.

Problem

• Transfer RL from simulation to real-world MAV control
• Handle complex dynamics of variable-pitch propeller MAVs
• Enable advanced aerial maneuvers like flips in reality

Innovation

• Real-to-sim transfer with system identification
• Domain randomization for robust simulations
• Cascade control for reliable deployment