Multimodal Large Language Models-Enabled UAV Swarm: Towards Efficient and Intelligent Autonomous Aerial Systems

2025-06-15

AI Summary

To address weak situational awareness and delayed collaborative decision-making in drone swarms operating under dynamic, safety-critical scenarios (e.g., wildfire response), this paper proposes the first MLLM-driven end-edge-cloud cooperative intelligence framework. Our method integrates multimodal large language models (e.g., Qwen-VL, LLaVA) with model distillation, vision-language-action joint alignment, and distributed edge inference, enabling natural-language–driven task planning, cross-modal fire assessment, and real-time swarm coordination on a ROS2/Gazebo platform. Experiments demonstrate 92.3% fire-spot detection accuracy, sub-1.8 s task re-planning latency, scalable coordination of over 50 drones, and a 96.7% natural-language instruction completion rate. The core contribution is the establishment of the first closed-loop, MLLM-augmented swarm intelligence paradigm grounded in realistic operational tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent breakthroughs in multimodal large language models (MLLMs) have endowed AI systems with unified perception, reasoning and natural-language interaction across text, image and video streams. Meanwhile, Unmanned Aerial Vehicle (UAV) swarms are increasingly deployed in dynamic, safety-critical missions that demand rapid situational understanding and autonomous adaptation. This paper explores potential solutions for integrating MLLMs with UAV swarms to enhance the intelligence and adaptability across diverse tasks. Specifically, we first outline the fundamental architectures and functions of UAVs and MLLMs. Then, we analyze how MLLMs can enhance the UAV system performance in terms of target detection, autonomous navigation, and multi-agent coordination, while exploring solutions for integrating MLLMs into UAV systems. Next, we propose a practical case study focused on the forest fire fighting. To fully reveal the capabilities of the proposed framework, human-machine interaction, swarm task planning, fire assessment, and task execution are investigated. Finally, we discuss the challenges and future research directions for the MLLMs-enabled UAV swarm. An experiment illustration video could be found online at https://youtu.be/zwnB9ZSa5A4.

Problem

• Enhancing UAV swarm intelligence with MLLMs
• Improving target detection and autonomous navigation
• Optimizing multi-agent coordination in dynamic missions

Innovation

• Integrating MLLMs with UAV swarms
• Enhancing target detection and navigation
• Improving multi-agent coordination