ARbiter: Generating Dialogue Options and Communication Support in Augmented Reality

2025-03-07

AI Summary

This study addresses the lack of real-time, multimodal assistance in everyday conversations. Methodologically, it pioneers a novel AR–AI dialogue support paradigm by systematically adapting video game dialogue mechanics to context-aware interfaces for AR glasses (e.g., XREAL, Meta Orion), implemented via a cloud-edge collaborative architecture integrating lightweight LLMs (o3-mini, DeepSeek-R1), ASR, and AR rendering modules. The contributions are threefold: (1) the first systematic transfer of gamified dialogue logic—such as branching narratives and contextual feedback—to natural AR–AI conversational settings; (2) the design and implementation of ARbiter, an extensible prototype framework that formalizes key research challenges and cross-scenario deployment pathways; and (3) bridging a critical gap in low-latency, context-adaptive, and linguistically natural dialogue support enabled by tight LLM–AR device integration—thereby establishing a new paradigm for intelligent assistance in social and professional communication.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

In this position paper, we propose researching the combination of Augmented Reality (AR) and Artificial Intelligence (AI) to support conversations, inspired by the interfaces of dialogue systems commonly found in videogames. AR-capable devices are becoming more powerful and conventional in looks, as seen in head-mounted displays (HMDs) like the Snapchat Spectacles, the XREAL glasses, or the recently presented Meta Orion. This development reduces possible ergonomic, appearance, and runtime concerns, thus allowing a more straightforward integration and extended use of AR in our everyday lives, both in private and at work. At the same time, we can observe an immense surge in AI development (also at CHI). Recently notorious Large Language Models (LLMs) like OpenAI's o3-mini or DeepSeek-R1 soar over their precursors in their ability to sustain conversations, provide suggestions, and handle complex topics in (almost) real time. In combination with natural language recognition systems, which are nowadays a standard component of smartphones and similar devices (including modern AR-HMDs), it is easy to imagine a combined system that integrates into daily conversations and provides various types of assistance. Such a system would enable many opportunities for research in AR+AI, which, as stated by Hirzle et al., remains scarce. In the following, we describe how the design of a conversational AR+AI system can learn from videogame dialogue systems, and we propose use cases and research questions that can be investigated thanks to this AR+AI combination.

Problem

• Combining AR and AI to enhance conversational support.
• Exploring AR+AI integration for everyday and workplace use.
• Designing AR+AI systems inspired by videogame dialogue interfaces.

Innovation

• Combines AR and AI for conversation support
• Uses AR-capable devices like HMDs
• Integrates LLMs for real-time dialogue assistance