The Impact of Surface Co-location and Eye-tracking on Mixed Reality Typing

2024-10-05

arXiv.org

AI Summary

This study addresses the low typing accuracy and frequent accidental activations encountered when using tactile-free virtual keyboards in mixed reality (MR). To mitigate these issues, we propose surface co-location—leveraging physical desktops or walls as implicit haptic references—and design an eye-gaze dwell-time–based anti-activation mechanism. Our method integrates egocentric hand tracking from consumer-grade head-mounted displays, real-time eye tracking, spatially anchored virtual keyboards, and a deterministic text input engine. Key contributions include: (1) the first application of surface co-location to replace explicit haptic feedback for virtual keyboard interaction; and (2) an eye-gaze–driven dynamic key activation strategy that significantly suppresses unintended keystrokes. Experimental results show a peak single-finger mid-air typing rate of 12 WPM; gaze-assisted ten-finger typing reduces backspace usage by 50%; and error rates remain statistically equivalent across conditions—demonstrating an effective trade-off between input efficiency and accuracy.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements

Abstract

Accuracy and speed are pivotal when typing. We hypothesized that the lack of tactile feedback on midair mixed reality keyboards may adversely impact performance. Our first experiment investigated providing tactile feedback to users typing in mixed reality by co-locating the virtual keyboard on a table or a wall. The keyboard was deterministic (without auto-correct), allowed mixed case typing with symbols, and relied only on the hand-tracking provided by a commodity headset's egocentric cameras. Users preferred and had the highest entry rate of 12 words-per-minute using a midair keyboard. Error rates were similar in all conditions. Based on user feedback, our second experiment explored ten-finger typing. We used a novel eye-tracking technique to mitigate accidental key presses. This technique was preferred for ten-finger typing and halved the number of times backspace was pressed. However, participants were faster using only their index fingers without eye-tracking at 11 words-per-minute.

Problem

• Investigate tactile feedback in mixed reality keyboards
• Compare typing performance with and without eye-tracking
• Evaluate user preference and error rates in midair typing

Innovation

• Surface co-location for tactile feedback
• Eye-tracking to reduce key press errors
• Hand-tracking using egocentric cameras