On the Use of Immersive Digital Technologies for Designing and Operating UAVs

2024-07-23

arXiv.org

AI Summary

UAVs face critical challenges in complex real-world environments, including misalignment between modeling and physical deployment, insufficient situational awareness, and delayed cyber-physical coordination. Method: This study proposes a generative AI (GAI)-driven closed-loop optimization framework integrating digital twin (DT), extended reality (XR), and deep reinforcement learning (DRL). It pioneers the systematic fusion of GAI to enable dynamic DT model evolution and semantic human–machine collaboration within XR environments, significantly enhancing real-time cyber-physical mapping fidelity and interaction depth. Contribution/Results: We establish an AI-augmented immersive UAV technical architecture that systematically addresses six key challenges, yielding a deployable technological pathway. The framework advances a novel paradigm for intelligent unmanned systems—grounded in both theoretical rigor and engineering feasibility—that enables highly adaptive, safe, and efficient autonomous mission execution.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Intelligent Robots: Embodied AI
• Computer Vision: Remote Sensing / Geospatial AI

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Unmanned Aerial Vehicles (UAVs) offer agile, secure and efficient solutions for communication relay networks. However, their modeling and control are challenging, and the mismatch between simulations and actual conditions limits real-world deployment. Moreover, improving situational awareness is essential. Several studies proposed integrating the operation of UAVs with immersive digital technologies such as Digital Twin (DT) and Extended Reality (XR) to overcome these challenges. This paper provides a comprehensive overview of the latest research and developments involving immersive digital technologies for UAVs. We explore the use of Machine Learning (ML) techniques, particularly Deep Reinforcement Learning (DRL), to improve the capabilities of DT for UAV systems. We provide discussion, identify key research gaps, and propose countermeasures based on Generative AI (GAI), emphasizing the significant role of AI in advancing DT technology for UAVs. Furthermore, we review the literature, provide discussion, and examine how the XR technology can transform UAV operations with the support of GAI, and explore its practical challenges. Finally, we propose future research directions to further develop the application of immersive digital technologies for UAV operation.

Problem

• Digital Twin
• Extended Reality
• Deep Reinforcement Learning

Innovation

• Digital Twin
• Generative AI
• Extended Reality