Survey of Large Language Models in Extended Reality: Technical Paradigms and Application Frontiers

2025-08-04

AI Summary

The intersection of large language models (LLMs) and extended reality (XR) lacks a unified technical framework and practical guidelines, hindering systematic advancement. Method: This study proposes the first taxonomy for LLM-augmented XR systems, structured around three pillars: interactive agent control, generative scene synthesis, and XR development toolkits. It integrates theoretical foundations with domain-specific applications—including immersive education, clinical healthcare, and industrial manufacturing—to distill design principles for intelligent XR experiences. Contribution/Results: The work identifies and analyzes core challenges—multimodal alignment, real-time constraints, and trustworthy interaction—and establishes foundational theory and actionable best practices for next-generation intelligent immersive systems that are interpretable, generative, and collaborative.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation, and their integration with Extended Reality (XR) is poised to transform how users interact with immersive environments. This survey provides a comprehensive review of recent developments at the intersection of LLMs and XR, offering a structured organization of research along both technical and application dimensions. We propose a taxonomy of LLM-enhanced XR systems centered on key technical paradigms -- such as interactive agent control, XR development toolkits, and generative scene synthesis -- and discuss how these paradigms enable novel capabilities in XR. In parallel, we examine how LLM-driven techniques support practical XR applications across diverse domains, including immersive education, clinical healthcare, and industrial manufacturing. By connecting these technical paradigms with application frontiers, our survey highlights current trends, delineates design considerations, and identifies open challenges in building LLM-augmented XR systems. This work provides insights that can guide researchers and practitioners in advancing the state of the art in intelligent XR experiences.

Problem

• Surveying integration of LLMs and XR for immersive interaction
• Classifying LLM-enhanced XR systems by technical paradigms
• Exploring LLM-driven XR applications across diverse domains

Innovation

• LLM-enhanced XR systems taxonomy
• Interactive agent control in XR
• Generative scene synthesis tools