Cooperative Multi-UAV Navigation in Complex Environments via Systematic Multi-Agent Deep Reinforcement Learning

2026-07-28

AI Summary

This work addresses key challenges in multi-UAV cooperative navigation within complex environments—namely, susceptibility to local optima, sparse rewards, imbalanced learning dynamics, and poor cross-scenario generalization. To overcome these issues, the authors propose a systematic multi-agent deep reinforcement learning framework that integrates a collaborative exploration mechanism leveraging memory and directional novelty, a hierarchical cooperative demonstration buffer, a safety-aware dual-conditioned curriculum scheduler, and a structure-aware policy generalization method grounded in local geometric features. By synergistically combining behavior cloning, curriculum learning, and a mixture-of-experts architecture, the approach significantly enhances collaborative success rates, navigation robustness, zero-shot transferability across unseen scenarios, and adaptability to dynamic perturbations in mixed static-dynamic obstacle environments.

Abstract

Cooperative navigation of multi-agent UAVs in complex environments faces key challenges including local optima traps, sparse rewards, learning imbalance among agents, and insufficient cross-scenario generalisation. This paper proposes a multi-agent deep reinforcement learning framework that addresses these issues through coordinated exploration, demonstration exploitation, safe curriculum scheduling, and structure-aware generalisation. First, a perception mechanism combining memory of visited states, directional novelty estimates, and penalty backpropagation enables agents to proactively detect and escape local optima. Second, a hierarchical collaborative demonstration buffer with tiered behaviour cloning manages trajectories by degree of team collaboration and applies differential supervision to the actor network, improving demonstration utilisation under sparse collaborative signals. Third, a safety-aware dual-condition curriculum scheduling mechanism reviews mastered scenarios through back-testing and experience pre-filling during training, suppressing catastrophic forgetting while ensuring both task performance and flight safety. For generalisation, local geometric features computed from sensor readings are abstracted into a domain parameter, through which a structure-aware gating network and mixture-of-experts mechanism condition the policy on local structural patterns rather than scenario-specific coordinates, enabling cross-scenario transfer without exposure to the target environment. The framework is further validated under mixed static-dynamic obstacle settings, showing robust adaptability to dynamic disturbances. Simulation results confirm strong performance in collaboration success rate, navigation robustness, zero-shot cross-scenario generalisation, and dynamic environment adaptability.

Problem

• cooperative navigation
• multi-UAV
• local optima
• sparse rewards
• cross-scenario generalisation

Innovation

• multi-agent deep reinforcement learning
• cooperative UAV navigation
• curriculum scheduling
• structure-aware generalisation
• hierarchical demonstration buffer