STAR: Smartphone-analogous Typing in Augmented Reality

2025-11-26

AI Summary

To address the lack of efficient, low-learning-cost text input methods in augmented reality (AR), this paper proposes STAR: a bare-hand skin-based two-thumb typing technique. STAR is the first to adapt the smartphone QWERTY thumb-typing paradigm to AR by integrating real-time hand pose tracking, virtual keyboard projection, and tactile feedback simulation—enabling a perceptible virtual keyboard directly on the user’s dorsal hand or palm surface while preserving existing typing habits. Its core innovation lies in a peripheral-free, non-occlusive, gesture-preserving input mechanism that leverages the skin as an interactive surface. After only 30 minutes of training, users achieved an average typing speed of 21.9 words per minute (56% of typical smartphone performance) with an error rate of just 0.3%, significantly outperforming state-of-the-art AR text input approaches.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Manipulation
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

While text entry is an essential and frequent task in Augmented Reality (AR) applications, devising an efficient and easy-to-use text entry method for AR remains an open challenge. This research presents STAR, a smartphone-analogous AR text entry technique that leverages a user's familiarity with smartphone two-thumb typing. With STAR, a user performs thumb typing on a virtual QWERTY keyboard that is overlain on the skin of their hands. During an evaluation study of STAR, participants achieved a mean typing speed of 21.9 WPM (i.e., 56% of their smartphone typing speed), and a mean error rate of 0.3% after 30 minutes of practice. We further analyze the major factors implicated in the performance gap between STAR and smartphone typing, and discuss ways this gap could be narrowed.

Problem

• Develops AR text entry method using virtual keyboard on hands
• Addresses performance gap between AR and smartphone typing speeds
• Leverages smartphone typing familiarity for augmented reality interfaces

Innovation

• Virtual QWERTY keyboard on hand skin
• Leverages smartphone two-thumb typing familiarity
• Achieves 21.9 WPM speed with low error rate